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Message from Local Initiative Support Corporation (LISC) 


To support and empower people who have been living on low incomes, helping them stabilize finances 
and take on good, family-sustaining jobs, is an evolving, and critical, science. At LISC, we developed our 
Financial Opportunity Centers® (FOCs) as part of that evolution, and research has shown that FOCs’ 
multi-pronged approach—offering employment services, financial coaching and access to income 
supports (like Supplemental Nutrition Assistance Program, rental assistance and child care subsidies) is 
good science. FOCs have supported 25,000 people each year in stabilizing their finances and joining the 
mainstream economy. 


But our experience has also shown us that FOC support, on its own, is not enough in all cases. We learned 
that many clients’ potential for long-term financial stability and wealth-building was stymied by the low- 
wage jobs available to them and by gaps in their education or the basic skills needed to succeed in the 
kinds of occupational training that can ultimately lead to a sustaining career. 


So we developed an additional strategy—Bridges to Career Opportunities (BCO)—which builds on our 
FOC model and which we began piloting seven years ago in six cities. In addition to the bundled services 
provided by FOCs, BCO clients receive instruction in basic skills using curricula tailored to occupations 
in each region’s growth industries—often with the partnership of local employers looking to fill jobs in 
those occupations. This foundational training, which includes soft skills instruction, is a doorway to more 
advanced training and certification that in turn leads to an occupational pathway with potential for upward 
mobility. The initial success of the BCO model inspired funders to support scaling the work, and we now 
have BCO programming through our FOCs in 40 locations across the country. 


Which brings us to this evaluation report. In 2017, Abt Associates began a study of the job and financial 
stability outcomes for BCO participants (involving some 1,100 clients) with follow-up assessment over 
the course of an additional 14 months, comparing their successes with those of people enrolled in FOC 
programming alone. The results were very positive, once again demonstrating that this common sense 
model for supporting people to prepare for and advance in 21*-century jobs is good science. The study 
found that: 


e BCO program participants had high completion rates (86%) and high credential attainment rates 
(80% of people who completed the program received credentials). 


e BCO participants showed a gain of one grade level in reading on the Test for Adult Basic 
Education (TABE®). 


e When it came to employment, BCO participants were more likely to have found or advanced in a 
job than their FOC counterparts — there was a difference of 19% between the two (56% vs 37%). 


The study also found that that certain financial outcomes, including credit score and net worth, were 
better for FOC participants than for BCO participants. This is likely due to the additional time FOC 
participants had when compared to BCO participants who spent more than three to six months in training. 
Nevertheless, credit outcomes were good for both BCO and FOC participants—80% of FOC comparison 
clients and 68% of BCO participants increased their credit score. We expect that over time, we would see 
similar outcomes for both cohorts. 


As encouraging as these outcomes are, this study, like all gauges of economic well-being in the country, 
were upended by the onset of the COVID-19 pandemic, and the staggering unemployment of upwards of 


30 million people last year. On top of the sweeping loss of jobs, one in three Americans took a cut in pay 
or hours. And it has been widely reported that unemployment and other indicators of the financial fallout 
from the pandemic have disproportionately devastated Black and Brown communities—the very 
communities that are home to the majority of FOC and BCO clients. 


It will take some time to fully grasp and quantify the pandemic’s impact on our FOC and BOC clients. 
Nevertheless, we can see, as noted in this report, that our strategies work and that the Bridges to Career 
Opportunities model provides imperative skills and training without which people simply cannot access 
quality jobs—and that will still be true in a post-pandemic work. As we continue to hone and test the very 
human science of promoting greater financial well-being for people who have lived in privation and 
instability, we will need these important results to guide us, and the people and communities we serve, 
along the way. 


PREFACE 


About This Report 


This report provides findings from Abt Associates’ (Abt) evaluation of the Local Initiatives Support 
Corporation’s (LISC) Bridges to Career Opportunities program. The evaluation was funded by the 
Corporation for National and Community Service’s Social Innovation Fund and supported by LISC. 
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EXECUTIVE SUMMARY 


Executive Summary 


Introduction 


This report presents the findings from Abt Associates’ (Abt) implementation study and impact study of 
the Bridges to Career Opportunities (BCO) model. BCO is a multi-component intervention that assists 
adults who are underemployed or unemployed in advancing in a career and obtaining a living wage. 
Created by the Local Initiatives Support Corporation (LISC), BCO has four components of services. The 
bridge component provides contextualized basic skills education, career planning, and occupational skills 
training. The other three components of BCO services compose LISC’s Financial Opportunity Center® 
(FOC) model, which bundles financial coaching and education, income support services, and employment 
services. The integrated bridge and FOC services—the BCO model—is intended to assist adults with low 
incomes or low skills earn industry-recognized credentials that can enable them to obtain jobs in career 
pathways leading to financial stability and economic well-being. 


With funding from the Corporation for National and Community Services (CNCS)’s Social Innovation 
Fund (SIF) in 2016, LISC implemented the BCO model in 32 community-based organizations in 11 
metropolitan areas. As part of the SIF project, LISC contracted with Abt to conduct an evaluation of the 
BCO model from 2016 to 2020. The evaluation consisted of an implementation study and a quasi- 
experimental impact study. The implementation study assessed the extent to which a purposeful sample of 
seven of the BCO treatment group sites carried out the services that compose the BCO model, and the 
factors that facilitated and posed challenges to delivering BCO services. The quasi-experimental impact 
study investigated the effects of BCO services in those seven sites on participants’ employment and 
financial well-being outcomes compared to those of a comparison group of similar clients who enrolled in 
FOC only services in six comparison sites. 


The evaluation’s treatment and comparison groups included 1,133 BCO participants and 1,217 FOC 
participants, respectively, who enrolled in those services during the period April 1, 2017 through 
December 31, 2018. The evaluation targeted a moderate level of evidence and followed the overall 
procedures that Abt submitted in its BCO SIF evaluation plan (Gan, Alamprese, Litwok, & Price, 2016). 


Context for the Evaluation and Related Research 


Families with low incomes face significant barriers to achieving financial security and upward mobility. 
Workers in low-wage occupations, particularly workers with a secondary education or less, tend to remain 
in these occupations (e.g., Gabe, Abel, &Florida, 2018; Andersson, Holzer, & Lane, 2005). 


To aid families in mitigating those barriers, LISC developed the Financial Opportunities Centers®. Based 
on the Annie E. Casey Foundation’s Center for Working Families model, the FOCs assist adults in 
working toward financial stability by advancing their employment and subsequently their financial well- 
being. LISC provides non-profit, community-based organizations with financial support and technical 
assistance to offer FOC bundled services in three areas: employment services; financial coaching and 
education; and linkages to income support services (e.g., Supplemental Nutrition Assistance Program, 
rental assistance, childcare subsidies). LISC’s goal is to increase the positive net income and, ultimately, 
financial stability for adults with low incomes through their participation in FOC services. 


Research on FOC clients has informed LISC’s ongoing approach to services. Early evidence from a study 
of FOC clients indicated that the combination of financial coaching and employment services increased 
FOC clients’ placement in jobs and job retention, as well as their net income. Financial coaching and 
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assistance in accessing relevant income support streams enabled FOC clients to stabilize their finances in 
preparing for and attaining placement in a job. However, FOC clients’ potential for long-term financial 
advancement was found to be limited by the low wage rates of jobs available to them and by their gaps in 
the basic skills needed to succeed in sector-based occupational training (Rankin, 2015). 


A quasi-experimental study of FOC client outcomes conducted by the Economic Mobility Corporation 
found that a key barrier to study participants’ financial stability is their ability to acquire “good” jobs— 
jobs that meet their expenses, have stable schedules, and have opportunities for long-term career growth 
(Roder, 2016). Though occupational training programs for in-demand jobs are available through the 
publicly funded workforce system, community colleges, and private organizations, many FOC clients 
have gaps in their basic skills, lack educational credentials, or have financial constraints that preclude 
their enrollment or success in those programs. 


To address challenges faced by adults with low skills and low incomes in obtaining family-sustaining jobs 
in career pathways, LISC considered the additional services that could strengthen adults’ basic and 
technical skills and attainment of credentials. In 2015 LISC launched Bridges to Career Opportunities, 
which was built on the FOC model and integrates bridge education and training services. 


BCO Model 


A key assumption underlying BCO is that the model’s components of services are necessary to enable 
adults to develop a career and earn a family-sustaining wage, and that those components are most 
effective when they work together (Exhibit 1). 


Exhibit 1. Integration of BCO Model’s Components 


Income 
Support 
Services 


Employment 
Services 


Financial 
Coaching and 
Education 


The bridge component increases career awareness and provides instruction in basic skills using 
occupationally contextualized curricula that can prepare adults to succeed in training for an occupational 
pathway. Its contextualized basic skills can be delivered as a first step to prepare BCO clients for 
occupational training or can be integrated into the occupational training. The development of clients’ 
basic skills within the context of work enables them to earn credentials and enter jobs with a potential for 
upward mobility. 
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BCO clients’ enrollment in public benefits can provide support for daily expenses and reduce barriers to 
training and employment. Financial coaching and education are expected to help clients reduce expenses, 
access credit under better terms, increase savings, and build assets. The assumption is that when clients 
are financially stable, they are more likely to engage in and remain in training. 


Finally, employment services ensure that clients are prepared for the job search and application processes, 
and that they are connected to opportunities for jobs with family-sustaining wages. 


Implementation Study 


Implementation Study Research Question 
The BCO implementation study addressed the following question: 


To what extent did the study sites implement the following four components of the BCO program with 
fidelity: (1) contextualized bridge program services, (2) financial coaching and education services, (3) 
income supports, and (4) employment services? 


Exhibit 2 lists the implementation study research question for each BCO component and associated 
activities. 


Implementation Study Design, Measures, and Data Collection 

Abt used a mixed methods design to assess the extent to which treatment sites’ delivery of the BCO 
model adhered to the characteristics of the model as defined by LISC. This approach involved having a 
well-defined set of BCO components of services and associated activities that include features of the 
structure and processes used in delivering the intervention and collecting data about the fidelity of 
implementation using multiple methods. 


The measures for BCO treatment sites’ fidelity of implementation of the BCO model are the activities in 
Exhibit 2 that are associated with each of the four BCO components—contextualized bridge services, 
financial coaching and education, income supports, and employment services. 


Data to address the overarching implementation study question To what extent did the treatment sites 
implement the BCO program with fidelity? were collected from the treatment sites using three methods: 
(1) interviews with evaluation sites’ staff as part of two rounds of site visits that Abt conducted; (2) 
observations of services delivered in the BCO treatment sites during the site visits; and (3) conduct of bi- 
annual interim telephone conferences during the two-year period of sites’ implementation of BCO 
services. 


Implementation Study Sites 

Abt selected seven BCO programs as treatment sites. These sites had strong capacity to deliver BCO 
services and sufficient projected baseline enrollment to meet the evaluation’s requirements. The sites 
were: 


e Brighton Center, Inc., Greater Cincinnati and Northern Kentucky; 
e Chinese Community Center, Inc., Houston, TX; 

e District 1199C Training and Upgrading Fund, Philadelphia, PA; 
e Instituto del Progreso Latina, Chicago, IL; 


e International Institute of Minnesota, St. Paul, MN; 
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e Project for Pride in Living, Inc., Minneapolis, MN; and 


e Wesley Community Center, Inc., Houston, TX. 


Exhibit 2. Implementation Study Research Questions for BCO Model Components 


Bridge Services Component: To what extent did BCO treatment sites implement bridge services and their associated 
activities? 


1. Instruction 

a. Addresses career pathway in growth sector or industry; 

b. Includes an academic pathway; 

c. Teaches basic skills that are contextualized to a specific career path; 
d. Is based on industry-specific curriculum; 

e. Enables clients to attain industry-recognized credentials; and 

f. Has measurable endpoint or clearly defined completion criteria. 


2. Assessment 
a. Uses criterion levels to determine client participation; and 
b. Asses pre- and post-test skills. 


3. Career Coaching 
a. Provides ongoing career coaching; and 
b. Connects academic and career pathways. 


Financial Coaching and Education Component: To what extent did the BCO treatment sites implement financial 


coaching and education services and their associated activities? 


a. Assesses clients’ financial health, combining a financial profile, budget, balance, and credit information (Combined 
Financial Assessment); 

b. Provides regular one-on-one interactions concerning clients’ financial status; 

c. Reviews clients’ credit reports every six months; and 

d. Uses financial products during coaching. 


Income Support Services Component: To what extent did the BCO treatment sites implement income support services 
and their associated activities? 


a. Screens for public benefits eligibility; 
b. Identifies income supports to enable participants to remain in training; and 
c. Reassesses clients’ needs as their situations change. 


Employment Services Component: To what extent did the BCO treatment sites implement employment services and 
their associated activities? 


a. Assists clients in preparing job search materials and conducting job search; 
b. Provides clients with coaching about job search, placement, retention, and advancement; and 
c. Supports clients in exploring careers. 
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Data Analysis 
The data collected through the site visits (interviews, observations) and the interim telephone conferences 


were analyzed to assess the overall fidelity of delivery of the BCO model. A content analysis was 
performed to understand the types of activities that the treatment sites conducted for each of the BCO 
components. 


The results from the content analysis of each treatment site’s BCO activities were rated to determine how 
closely a treatment site had adhered to the BCO model, defined as whether the four components of the 
BCO model were delivered as intended. Abt developed a three-point scale to rate the degree to which 
each of the activities for each component was implemented as specified in the BCO list of components. 


Findings on Fidelity of BCO Implementation 
The seven treatment sites demonstrated high fidelity in implementing the BCO model’s four components 


of activities “as intended;” that is, the sites carried out the activities associated with each of the BCO 
components as they have been defined by LISC. Overall, the treatment sites implemented 94 percent of 
the activities across the four components “as intended,” with little variation in the percentages of 
adherence for each component. The component with the highest percentage of adherence was 
employment services (95.1%), followed by financial coaching and education services, along with income 
support services (94.1%), and bridge services (93.6%). Those results indicate that BCO sites participating 
in the evaluation were able to follow the guidance and assistance that LISC provided in offering bundled 
services to support adults with low incomes. 


The treatment sites were able to achieve a high degree of adherence to the BCO model while using a 
range of approaches in carrying out the activities associated with each of the BCO components. This 
variation reflected the resources of the organizations implementing BCO and their prior experience as 
Financial Opportunity Centers®, which provided a foundation for their approach to BCO services. 
Another factor influencing the BCO sites’ services were the types of partnerships with external 
organizations that staff formed to enable them to provide comprehensive services to clients with low 
incomes or low skills. Those partnerships were important in enabling BCO sites to leverage their services 
to meet the varying needs of their clients. The background and experience of staff affected the treatment 
sites’ approaches to bridge, financial coaching, and education services since those services require 
specialized knowledge and skills to be able to work effectively with clients. The variation in approaches 
points to the flexibility within the BCO model and how the model is used by different types of 
organizations. 


Impact Study 


The impact study examined whether the BCO model, which integrates bridge services with FOC services, 
better prepares participants with low incomes to increase their employment prospects and financial 
stability than do FOC services alone. The study addressed research questions on employment and 
financial well-being using a quasi-experimental design and compared the outcomes of BCO participants 
(the treatment group) with a matched group of FOC participants (the comparison group). 


Impact Study Questions 
The confirmatory research questions for the impact study were: 


e Were BCO participants more likely to obtain or advance in a job after program entry relative to a 
comparison group who received only FOC services? (Employment) 
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e Were BCO participants more likely to have obtained or increased their credit scores after program 
entry relative to a comparison group who received only FOC services? (Financial Well-Being) 


Sample 

as BCO programs were purposively selected as treatment sites, as described above, and six FOC 
programs were purposively selected as comparison sites. Those FOC sites did not provide educational 
services for clients; were in cities with similar labor markets as the treatment sites and served FOC clients 
with similar demographic characteristics as the BCO treatment sites; and collected the evaluation’s 
required data for FOC participants and delivered FOC services according to the FOC model. 


Across the BCO treatment programs and the FOC comparison programs, eligible BCO participants were 
exactly matched to one or more eligible FOC participant on gender, age, race/ethnicity, highest level of 
education completed, employment status, and credit score at intake using a coarsened exact matching 
(CEM) process. 


Impact Study Data Sources 

The data sources were extant administrative data that LISC maintains on BCO and FOC clients’ 
demographic and background information, service receipt, and outcome data. BCO and FOC sites collect 
and submit those data to LISC using standardized procedures. The data are stored in LISC’s database that 
uses a Salesforce platform. 


The data to address the impact study’s confirmatory research question concerning BCO and FOC-only 
participants’ job advancement were drawn from participants’ intake, employment, and advancement 
records in Salesforce. The data for the confirmatory research question concerning study participants’ 
credit score increase were drawn from the credit score record in Salesforce. 


Analytic Approach 
Participant outcomes were analyzed in a linear regression model that tested the effect of BCO on job 
advancement and credit score increase. 


Impact Study Results for Confirmatory Research Questions 


Abt’s quasi-experimental impact study addressed confirmatory research questions concerning impacts on 
study participants’ employment and financial well-being. 


Employment Impacts 

The evaluation’s confirmatory employment question was: Were BCO participants more likely to have a 
job improvement after program entry relative to a comparison group who received only FOC services? To 
address this question, Abt analyzed treatment and comparison group participants’ job improvement! 
during the 14 months after their enrollment in a BCO program or an FOC. The analyses indicated that 
study participants enrolled in BCO programs were 19 percentage points more likely to have a “job 
improvement” than were participants in FOC programs (56 percent versus 37 percent). This difference 
was Statistically significant at the .001 level. 


Participant (a) enrolled in the program unemployed and starts a new job in the 1-14 months after enrollment or (b) was 


employed at enrollment and has one of the following outcomes in the 1-14 months after enrollment: (1) obtains a new job 
and the new wage is greater than the wage at enrollment; or (2) has an increase in wages or obtains a promotion. 
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The BCO model includes activities that support program participants’ selection of a career pathway and 
reinforce participants’ advancement in a pathway, which might have contributed to the positive impacts 
on the treatment group participants’ job advancement. The BCO bridge services include an ongoing 
review of participants’ career goals and selection of pathways that align with their skills, interests, and 
abilities. When BCO participants earn an initial credential, they are encouraged to consider the steps they 
will take to obtain an advanced credential that will enable them to move forward in a career pathway. As 
participants work with employment coaches in job search and job placement activities, coaches discuss 
strategies for retaining and advancing in a job. The BCO model is designed to provide program 
participants with ongoing support and encouragement during a sustained period that can prepare them for 
success in the workplace. 


Financial Well-Being Impacts 

The evaluation’s confirmatory financial well-being question was: Were BCO participants more likely to 
have obtained or increased their credit scores after program entry relative to a comparison group who 
received only FOC services? To address this question, Abt analyzed treatment and comparison group 
participants’ maximum credit score at follow-up during the 14 months after their enrollment in a BCO 
program or an FOC. The analyses indicated that 68 percent of participants in the BCO program and 
approximately 80 percent of participants in the FOC program had an increase in their credit score after 
program entry. This negative impact of 18 percentage points was not statistically significant. 


One condition that might explain the credit score findings is the difference in timeline between BCO 
participants’ and FOC participants’ job search and placement. BCO participants are enrolled in bridge 
services involving concurrent or sequential occupational training for two to seven months after program 
enrollment and search for jobs in their area of occupational training after completing training. In contrast, 
FOC participants generally are placed in jobs as soon as possible after program entry. The earlier job 
placement for FOC participants could result in better credit scores for them during the 14 months after 
their enrollment in an FOC, because they will have begun working sooner after enrollment than BCO 
participants. 


Another factor that might have affected the credit score analysis is the small sample of matched BCO 
participants (181) and FOC participants (79) who had baseline and follow-up credit scores. Although the 
BCO and FOC programs are strongly encouraged to retrieve program participants’ credit scores every six 
months during their engagement in services, there was a high percentage of missing credit score data, 
particularly for FOC participants. The FOC participants’ missing data might reflect their shorter length of 
time receiving services compared with BCO participants. 


Since the BCO model is one of the few career pathways programs that includes a financial well-being 
intervention and data collection activities, we are not able to examine the financial well-being results of 
this study relative to the results from other studies. 


Descriptive Analyses of Treatment Group’s Education Outcomes 


Abt conducted descriptive analyses of the BCO treatment group participants’ education outcomes to 
understand participants’ progress in developing their foundational skills while enrolled in BCO, in 
completing the BCO program, and their progress in earning educational credentials. The evaluation’s 
confirmatory education question could not be addressed due to lack of data and is discussed below in the 
section on “Changes to SEP.” 
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BCO Participants’ Foundational Literacy Skills 

As part of BCO’s bridge services, BCO treatment sites were to provide contextualized instruction to 
increase the foundational literacy skills of BCO participants who had low basic skills. To examine the 
effects of BCO on literacy skills, we compared the pre- and post-test scores on the Test for Adult Basic 
Education (TABE®) Reading test for all BCO participants in the evaluation, not for just those with low 
basic skills at entry. 


At BCO program intake, BCO treatment group participants’ TABE® Reading pre-test scale score was 
532, equivalent to about the 6.6 grade level. At the post-test, BCO participants had increased their reading 
scores by about 9 points or by about one grade-level to the 7.4 grade-level equivalent (CTB McGraw-Hill, 
2004). This increase was statistically significant at p< .001 level. 


BCO Participants Program Completion and Credential Attainment 

Abt’s evaluation examined the treatment group participants’ rates of completion of a BCO program and 
attainment of educational credentials. The majority (86%) of the treatment group participants completed 
the BCO education or training program in which they enrolled. Of treatment group participants who 
completed a training program, approximately 80% percent earned at least one credential, primarily an 
occupational certificate. Over half (57%) of those credentials were in healthcare occupations, which was 
the occupational focus of over half of the education and training programs offered by BCO treatment 
sites. 


BCO treatment group participants’ rates of program completion and credential attainment by month 14 
after enrollment compare favorably with the results from education and training programs funded by the 
U.S. Department of Health and Human Services Health Professional Opportunity Grants (HPOG). HPOG 
grants are intended to assist adults with low incomes earn healthcare credentials and obtain family- 
sustaining jobs in a pathway. The HPOG data provide a context for considering the BCO treatment sites’ 
outcomes in training program completion and credential attainment. An evaluation of Pima Community 
College’s Pathways to Healthcare Program, which includes basic skills bridge programs, found that by 
month 18 after program enrollment, 59 percent of participants had completed any education or training 
and 29 percent had received a credential (Gardiner, Rolston, Fein, & Cho, 2017). An evaluation of San 
Diego County Bridge to Employment in the Healthcare Industry Program found that 64 percent of the 
treatment group members earned a credential (Farrell & Martinson, 2017). 


Changes to SEP 


Abt made changes to the SEP primarily because of the lack of availability of evaluation participants’ 
administrative data. The changes to the SEP are described below. 


Education Research Questions 
The confirmatory research question in education, “Are BCO participants more likely to have attained an 


occupational certificate or postsecondary credential 12 months after program entry relative to a 
comparison group who receive only FOC services?” could not be addressed due to lack of reliable 
comparison group data. Abt’s analysis of educational outcomes revealed than only four comparison 
participants had data concerning attainment of occupational certificates or postsecondary credentials at 
follow-up. The response rate might have been due to the structure of the Degree, Certificate, License 
record in LISC’s Salesforce database, in which program staff enter data in this record only if participants 
have attained a credential. There is no variable in the record concerning non-attainment of a credential, 
which meant we could not determine whether the response rate of four participants was accurate. 
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The exploratory research question in education for only treatment group members “Are BCO participants 
likely to increase their math skills (as measured by a standardized test) after participating in the BCO 
program?” could not be addressed because most treatment programs did not collect math post-test data 
from participants at the end of bridge services. Rather, the programs collected post-test data on 
participants’ reading skills since most instruction addressed the reading skills participants needed to 
succeed in their occupational training. 


The exploratory research question in education “Are BCO participants likely to complete their BCO 
program?” was added to the evaluation since it addressed an important hypothesized outcome from 
participation in BCO bridge services as indicated in the BCO logic model. 


Employment Research Questions 

Approach to Research Questions. Abt’s intended approach to address employment outcomes was to 
compare the treatment and comparison groups’ outcomes at 12 months after participants’ enrollment in 
BCO services (treatment group) and FOC services (comparison group). The timeline assumed that 
program staff would conduct follow-up data collection 12 months after participants enrolled in their 
respective programs. However, because the BCO model allows for client-driven interactions, only a 
portion of the follow-up data collection occurred at 12 months after enrollment. Because the BCO and 
FOC program services dictate that clients reach out to staff as needed, this interaction does not follow a 
specific schedule. As a result, it was difficult to gather point-in-time data according to a 12-month follow- 
up schedule. Rather, follow-up data were defined as any data collected after baseline, beginning in month 
1 after enrollment and continuing throughout the 14-month period in which the evaluation tracked clients’ 
participation in BCO or FOC services. BCO and FOC staff also reported that part of the variation in the 
timeline for follow-up data collection was due to sites’ difficulty in reaching clients after they ended their 
participation in BCO or FOC services. Thus the 12-month timeframe for the employment research 
questions was changed to the timeframe of month 1 through month 14. 


Addition of Employment Confirmatory Question. Prior to data analysis, Abt added the confirmatory 
research question “Were BCO participants more likely to obtain or advance in a job after program entry 
relative to a comparison group who received only FOC services?” based on our re-examination of the 
BCO logic model concerning participants’ outcomes from employment services. 


Exploratory Employment Questions Not Addressed. Abt could not address the following two 
employment exploratory questions due to the difficulty in gathering the point-in-time information noted 
above: (1) Are BCO participants more likely to retain the same job (for 180 days) relative to a comparison 
group who receive only FOC services? and (2) Are BCO participants likely to have been employed more 
quickly relative to a comparison group who receive only FOC services? 


Financial Well-Being Research Questions 
Approach to Research Questions. The intended approach and change in timeframe described for 
employment questions also applies to the financial well-being questions. 


Addition of Financial Well-Being Confirmatory Question. Prior to data analysis, Abt added the 
confirmatory research question “Were BCO participants more likely to have obtained or increased their 
credit scores after program entry relative to a comparison group who received only FOC services? based 
on our re-examination of the BCO logic model concerning participants’ outcomes from financial services. 


Addition of Financial Well-Being Exploratory Question. The following exploratory research question 
was added because of the emphasis on net-worth in financial coaching services: Were BCO participants 
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more likely to have increased their net worth after program entry relative to a comparison group who 
received only FOC services? 


Exploratory Financial Well-Being Questions Not Addressed. Abt could not address the following 
exploratory financial well-being question due to a lack of data on on-time account payments: Are BCO 
participants more likely to have a greater percentage of on-time account payments 12 months after 
program entry relative to a comparison group who receive only FOC services? 


Other Changes 

The SEP specified that two site visits would be conducted to each of the treatment programs as part of the 
implementation study. Abt conducted two site visits to six of the seven programs. Due to a scheduling 
difficulty, Abt could not conduct a site visit to the seventh program but conducted telephone interviews 
with program staff to gather the same information that was to be collected during the second site visit. 


The SEP specified that the evaluation would use a Propensity Score Analysis (PSA), a matching 
technique wherein participants in a program are matched to non-participants based on their participation 
propensity score. PSA is useful for selecting comparison members on a large number (20 or more) of 
characteristics. Given the small number of variables that were appropriate to use for matching in the BCO 
evaluation, Abt used the CEM approach instead of a PSA matching approach. Because all matching 
variables are combined into a summary score, PSA does not guarantee close or exact matches on 
variables of interest. However, CEM ensures exact matches on variables of interest (because treatment 
and comparison matches within the same block have identical values on coarsened matching variables). 


The SEP specified that the timeline for data collection for the impact study would be January 2017 
through December 2018. The beginning of the data collection period was changed from January 2017 to 
April 2017 because LISC’s training of programs’ implementation of their new Salesforce data system was 
still underway at the beginning of 2017. The change to April 2017 ensured that all BCO and FOC 
programs would be able to enter baseline client data into Salesforce at the beginning of the evaluation’s 
data collection period. 


Conclusions 


The evaluation found that the BCO model, with well-defined services and training and technical 
assistance support, can be implemented by community-based organizations with strong adherence to the 
model. The evaluation also indicated that BCO’s goal to facilitate clients’ attainment of educational 
credentials and employment on their path to economic well-being can be attained through the delivery of 
the multi-component services that define the BCO model. Although BCO evaluation participants were not 
able to make significant advances in their economic status during the 14 months after they enrolled in the 
BCO program, they were able to earn credentials and advance in their jobs. Those outcomes are thought 
to be steps toward economic well-being. The process of increasing individuals’ economic status is 
complex and affected by many factors. Although this evaluation provided preliminary insights about near- 
term changes in participants’ economic status, more robust investigations with larger samples of 
participants are needed to understand adults’ attainment of economic well-being. 


Fidelity of BCO Implementation 

The seven treatment sites implemented the BCO model’s four components of activities with a high degree 
of fidelity. Many factors contributed to this result including the ongoing training and technical assistance 
that LISC staff provided to its grantees and the organizational resources of the seven programs that 
participated as treatment sites. Another factor is that the BCO model is specified with sufficient detail to 
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guide programs’ main activities in each of the four BCO components but is not so prescriptive that 
programs cannot adjust services as they encounter challenges. This model design enabled the treatment 
programs to respond to the needs of their client populations while delivering the range of services that 
compose the BCO model. 


Education Outcomes 
The majority of BCO treatment participants completed the BCO program (86%) and most of those 


completers earned at least one credential, which was primarily an occupational certificate (80%). The 
treatment sites’ approaches to recruitment and orientation helped clients understand the requirements and 
commitment needed to succeed in the BCO program. The BCO bridge design also provided clients with 
an opportunity to master skills in the bridge component that could contribute to their success in the 
occupational training. The ongoing support provided by BCO coaches and instructors helped clients to 
persevere and keep on track as they worked to earn an educational credential. 


Employment Outcomes 
The evaluation results indicated that BCO treatment participants were significantly more likely to obtain 


or advance in a job after enrollment than FOC comparison participants. Factors contributing to this 
outcome were the comprehensive job readiness and job placement services that the BCO sites provided. 
As the BCO implementation study indicated, the treatment sites’ employment services were the most 
highly aligned of the four BCO components of services such that BCO sites provided many similar 
services. Another factor likely contributing to the BCO participants’ job placement and advancement was 
the partnerships that BCO sites formed with local employers. BCO staff reported that those partnerships 
not only facilitated participants’ job attainment but also help support participants as they moved along in 
their employment. 


Financial Well-Being Outcomes 

The evaluation found that FOC comparison group participants were more likely to increase their credit 
scores than BCO treatment group participants. Comparison group participants also were more likely than 
treatment participants to decrease their debt, increase their net worth, and have a higher amount of debt 
decrease. Comparison group participants’ earlier participation in the workforce after program enrollment 
may have enabled those participants to engage in financial activities helped to increase their financial 
well-being. 


Lessons Learned and Limitations 


One lesson from the evaluation concerns processes to mitigate the loss of administrative data. Although 
LISC and Abt encouraged sites throughout the evaluation to follow established protocols for entering 
client data, more frequent monitoring of the BCO and FOC data in Salesforce might have resulted in less 
loss of data. 


Because of the evaluation’s quasi-experimental design, the BCO treatment programs participating in the 
evaluation were not selected to represent the full set of BCO programs but were purposively selected 
because of their planned services, client population, stability of BCO program implementation, and 
projected number of participants. The FOC comparison programs were purposively selected for the 
evaluation because their clients who entered services during the evaluation’s enrollment period had 
similar background characteristics as the BCO clients in the treatment group. The BCO and FOC 
programs were also in cities with similar labor markets. For those reasons, the evaluation’s results may 
not be applicable to clients with background characteristics different from the backgrounds of evaluation 
participants or because of possible differences in BCO program services. 
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The goal of using coarsened exact matching (CEM) in the evaluation was to create treatment and 
comparison groups that were similar, before the start of clients’ participation in the BCO and FOC 
programs, on observed characteristics likely to be related to the education, employment, and financial 
outcomes of interest. Therefore, any differences in education, employment, and financial outcomes 
observed can be attributed to the BCO program (or unobserved differences) rather than to initial 
differences between the groups. The treatment and comparison groups’ analytic samples were shown to 
be equivalent at baseline on observed characteristics, even after the loss of sample due to missing 
outcome data. However, a limitation of the study design is that differences observed may be due to 
unobserved characteristics of participants or the BCO program. Further, because the missing data led to 
small sample sizes for the financial outcome, the evaluation may not be powered to detect small 
differences between the groups. 
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Introduction 


This report presents the findings from Abt Associates’ (Abt) implementation study and impact study of 
the Bridges to Career Opportunities (BCO) model. BCO is a multi-component intervention that assists 
adults who are underemployed or unemployed adults in advancing in a career and obtaining a living 
wage. Created by the Local Initiatives Support Corporation (LISC), BCO has four components of 
services. The bridge component provides contextualized basic skills education, career planning, and 
occupational skills training. The other three components of BCO services compose LISC’s Financial 
Opportunity Center® (FOC) model, which bundles financial coaching and education, income support 
services, and employment services. The integrated bridge and FOC services—the BCO model—is 
intended to assist adults with low incomes or low skills earn industry-recognized credentials that can 
enable them to obtain jobs in career pathways leading to financial stability and economic well-being. 


With funding from the Corporation for National and Community Services (CNCS)’s Social Innovation 
Fund (SIF) in 2016, LISC implemented the BCO model in 32 community-based organizations in 11 
metropolitan areas. As part of the SIF project, LISC contracted with Abt to conduct an evaluation of the 
BCO model from 2016 to 2020. The evaluation consisted of an implementation study and a quasi- 
experimental impact study. The implementation study assessed the extent to which a purposeful sample of 
seven of the BCO treatment group sites carried out the core services that compose the BCO model, and 
the factors that facilitated and posed challenges to delivering BCO services. The quasi-experimental 
impact study investigated the effects of BCO core services in those seven sites on participants’ 
employment and financial well-being outcomes compared to those of a comparison group of similar 
clients who enrolled in only FOC services in six comparison sites. 


The evaluation’s treatment and comparison groups included 1,133 BCO participants and 1,217 FOC 
participants, respectively, who enrolled in those services during the period April 1, 2017 through 
December 31, 2018.” The evaluation targeted a moderate level of evidence and followed the overall 
procedures that Abt submitted in its BCO SIF evaluation plan (Gan, Alamprese, Litwok, & Price, 2016). 


Context for the Evaluation and Related Research 


Families with low incomes face significant barriers to achieving financial security and upward mobility. 
Workers in low-wage occupations, particularly workers with a secondary education or less, tend to remain 
in these occupations (e.g., Gabe, Abel, &Florida, 2018; Andersson, Holzer, & Lane, 2005). Evidence 
suggests this is partially due to gaps in their technical skills and basic skills. For example, three-quarters 
of workers with low wages have a high school diploma or equivalent but lack the relevant occupational 
skills and employer connections required to enter a career pathway (Loprest, Acs, Ratcliffe, & Vinopal, 
2009). Upward mobility for those who lack technical skills, high school credentials, or basic skills is even 
more limited. 


To aid families in mitigating these barriers, LISC developed the Financial Opportunities Centers”. Based 
on the Annie E. Casey Foundation’s Center for Working Families model, the FOCs assist adults in 


2 SEP Change: The SEP specified that the timeline for data collection for the impact study would be January 2017 through 
December 2018. The beginning of the data collection period was changed from January 2017 to April 2017 because LISC’s 
training of programs’ implementation of their new Salesforce data system was still underway at the beginning of 2017. The 
change to April 2017 ensured that all BCO and FOC programs would be able to enter baseline client data into Salesforce at 
the beginning of the evaluation’s data collection period. 
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working toward financial stability by advancing their employment and subsequently their financial well- 
being. LISC provides non-profit, community-based organizations with financial support and technical 
assistance to offer FOC bundled services in three areas: employment services; financial coaching and 
education; and linkages to income support services (e.g., Supplemental Nutrition Assistance Program, 
rental assistance, childcare subsidies). LISC’s goal is to increase the positive net income and, ultimately, 
financial stability for adults with low incomes through their participation in FOC services. 


Research on FOC clients has informed LISC’s ongoing approach to services. Early evidence from a study 
of FOC clients indicated that the combination of financial coaching and employment services increased 
FOC clients’ placement in jobs and job retention, as well as their net income. Financial coaching and 
assistance in accessing relevant income support streams enabled FOC clients to stabilize their finances in 
preparing for and attaining placement in a job. However, FOC clients’ potential for long-term financial 
advancement was found to be limited by the low wage rates of jobs available to them and by their gaps in 
the basic skills needed to succeed in sector-based occupational training (Rankin, 2015). 


A quasi-experimental study of FOC client outcomes conducted by the Economic Mobility Corporation 
found that a key barrier to study participants’ financial stability is their ability to acquire “good” jobs— 
jobs that meet their expenses, have stable schedules, and have opportunities for long-term career growth 
(Roder, 2016). Though occupational training programs for in-demand jobs are available through the 
publicly funded workforce system, community colleges, and private organizations, many FOC clients 
have gaps in their basic skills, lack educational credentials, or have financial constraints that preclude 
their enrollment or success in these programs. 


To address challenges faced by adults with low skills and low incomes in obtaining family-sustaining jobs 
in career pathways, LISC considered the additional services that could strengthen adults’ basic and 
technical skills and attainment of credentials. In 2015 LISC launched Bridges to Career Opportunities, 
which was built on the FOC model and integrates bridge education and training services. 


BCO Logic Model 
The BCO logic model is presented in Appendix A. The BCO model provides clients with the following: 


e Bridge services that include contextualized adult basic education (ABE) or English as a Second 
Language (ESL) instruction to develop clients’ proficiencies in reading, writing, numeracy, and 
English language to enter and succeed in technical skills training; and connections to 
occupational training that lead to industry-recognized credentials and career pathways in locally 
in-demand occupations; and 


e Financial coaching and education services, income support services, and employment services 
that are integrated to help clients manage the financial, navigational, and logistical challenges 
associated with participation in education and training programs. 


A key assumption underlying BCO is that the model’s components of services are necessary to enable 


adults to develop a career and earn a family-sustaining wage, and that those components are most 
effective when they work together (Exhibit 1). 
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Exhibit 1. Integration of BCO Model’s Components 
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The bridge component increases career awareness and provides instruction in basic skills using 
occupationally contextualized curricula that can prepare adults to succeed in training for an occupational 
pathway. Its contextualized basic skills can be delivered as a first step to prepare BCO clients for 
occupational training or can be integrated into the occupational training. The development of clients’ 
basic skills within the context of work enables them to enter jobs with a potential for upward mobility. 


BCO clients’ enrollment in public benefits can provide support for daily expenses and reduce barriers to 
training and employment. Financial coaching and education are expected to help clients reduce expenses, 
access credit under better terms, increase savings, and build assets. The assumption is that when clients 
are financially stable, they are more likely to engage in and remain in training. 


Finally, employment services ensure that clients are prepared for the job search and application processes, 
and they are connected to opportunities for jobs with family-sustaining wages. 


Research Questions 


Implementation Study Research Questions 
The BCO implementation study research question was the following: 


To what extent did the study sites implement the following four components of the BCO program with 
fidelity: (1) contextualized bridge program services, (2) financial coaching and education services, (3) 
income supports, and (4) employment services? 


The research questions for the implementation study are shown in Exhibit 2. 
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Exhibit 2. Implementation Study Research Questions for BCO Model Components 


Bridge Services Component: To what extent did BCO treatment sites implement bridge services and their associated 
activities as defined by LISC? 


1. Instruction 
a. Addresses career pathway in growth sector or industry; 
Includes an academic pathway; 
Teaches basic skills that are contextualized to a specific career path; 
Is based on industry-specific curriculum; 
Enables clients to attain industry-recognized credentials; and 
Has measurable endpoint or clearly defined completion criteria. 


2. Assessment 
a. Uses criterion levels to determine client participation; and 
b. Assesses pre- and post-test skills. 


3. Career Coaching 
a. Provides ongoing career coaching; and 
b. Connects academic and career pathways. 


Financial Coaching and Education Component: To what extent did the BCO treatment sites implement financial 
coaching and education services and their associated activities as defined by LISC? 


Assesses Clients’ financial health, combining a financial profile, budget, balance, and credit information 
(Combined Financial Assessment); 

Provides regular one-on-one interactions concerning clients’ financial status; 

Reviews clients’ credit reports every six months; and 

Uses financial products during coaching. 


Income Support Services Component: To what extent did the BCO treatment sites implement income support services 
and their associated activities as defined by LISC? 


a. Screens for public benefits eligibility; 
b. Identifies income supports to enable participants to remain in training; and 
c. Reassesses clients’ needs as their situations change. 


Employment Services Component: To what extent did the BCO treatment sites implement employment services and 
their associated activities as defined by LISC? 


a. Assists clients in preparing job search materials and conducting job search; 
b. Provides clients with coaching about job search, placement, retention, and advancement; and 
c. Supports clients in exploring careers. 


Abt Associates BCO Evaluation Final Report April 2021 1 4 


INTRODUCTION 


Impact Study Research Questions 

The BCO quasi-experimental impact study investigated whether the BCO intervention, which integrates 
bridge services with FOC services, better prepares adults with low incomes and low skills to increase 
their employment prospects and financial stability than do FOC services alone. The impact study 
addressed confirmatory and exploratory research questions concerning study participants’ employment 
and financial well-being outcomes. The impact study also examined exploratory questions concerning 
BCO treatment group participants’ development of reading skills and occupational training completion. 
The impact study research questions are shown in Exhibit 3. 


Exhibit 3. Impact Study Research Questions 


Employment 
Abt addressed the following confirmatory and exploratory questions related to employment outcomes: 


Confirmatory Question: 
1. Were BCO participants more likely to obtain or advance in a job after program entry relative to a comparison 
group who received only FOC services? 


Exploratory Questions: 
2. Were BCO participants more likely to increase their hourly wages after program entry relative to a comparison 
group who received only FOC services? 


Were BCO participants more likely to be placed in jobs with higher wages relative to a comparison group who 
received only FOC services? 


Were BCO participants more likely to work more hours per week after program entry relative to a comparison 
group who received only FOC services? 


Financial Well-Being 
Abt addressed the following confirmatory and exploratory questions related to financial well-being outcomes: 


Confirmatory Question: 
1. Were BCO participants more likely to have obtained or increased their credit scores after program entry relative 
to a comparison group who received only FOC services? 


Exploratory Questions: 
2. Were BCO participants more likely to have lower overall debt after program entry relative to a comparison group 
who received only FOC services? 


Were BCO participants more likely to have increased their net worth after program entry relative to a comparison 
group who received only FOC services? 


Education 
Abt addressed descriptive research questions concerning BCO participants’ education outcomes: 


Are BCO participants likely to increase their reading skills (as measured by a standardized test) after 
participating in the BCO program? 


Are BCO participants likely to complete their BCO program? 


Are BCO participants likely to earn credentials after participating ina BCO program? 


Abt Associates BCO Evaluation Final Report April 2021 1] 5 


INTRODUCTION 


Changes to SEP Research Questions 


Abt made changes to the SEP primarily because of the lack of availability of evaluation participants’ 
administrative data. The changes to the research questions in the SEP are described below. 


Education Research Questions 

The confirmatory research question in education, “Are BCO participants more likely to have attained an 
occupational certificate or postsecondary credential 12 months after program entry relative to a 
comparison group who receive only FOC services?” could not be addressed due to lack of reliable 
comparison group data. Abt’s analysis of educational outcomes revealed than only four comparison 
participants had data concerning attainment of occupational certificates or postsecondary credentials at 
follow-up. The response rate might have been due to the structure of the Degree, Certificate, License 
record in LISC’s Salesforce database, in which program staff enter data in this record only if participants 
have attained a credential. There is no variable in the record concerning non-attainment of a credential, 
which meant we could not determine whether the response rate of four participants was accurate. 


The exploratory research question in education for only treatment group member “Are BCO participants 
likely to increase their math skills (as measured by a standardized test) after participating in the BCO 
program?” could not be addressed because most treatment programs did not collect post-test data from 
participants at the end of bridge services. Rather, the programs collected post-test data on participants’ 
reading skills since most instruction addressed the reading skills participants needed to succeed in their 
occupational training. 


The exploratory research question “Are BCO participants likely to complete their BCO program?” was 
added to the evaluation since it addressed an important hypothesized outcome from participation in BCO 
services as indicated in the BCO logic model. 


Employment Research Questions 

Approach to Research Questions. Abt’s intended approach to address employment outcomes was to 
compare the treatment and comparison groups’ outcomes at 12 months after participants’ enrollment in 
BCO services (treatment group) and FOC services (comparison group). The timeline assumed that 
program staff would conduct follow-up data collection 12 months after participants’ enrollment in their 
respective programs. However, because the BCO model allows for client-driven interactions, only a 
portion of the follow-up data collection occurred at 12 months after enrollment. Because the BCO and 
FOC program services dictate that clients reach out to staff as needed, this interaction does not follow a 
specific schedule. As a result, it was difficult to gather point-in-time data according to a 12-month follow- 
up schedule. Rather, follow-up data were defined as any data collected after baseline, beginning in month 
1 after enrollment and continuing throughout the 14-month period in which the evaluation tracked clients’ 
participation in BCO or FOC services. BCO and FOC staff also reported that part of the variation in the 
timeline for follow-up data collection was due to sites’ difficulty in reaching clients after they ended their 
participation in BCO or FOC services. Thus the 12-month timeframe for the employment research 
questions was changed to the timeframe of month | through month 14. 


Addition of Employment Confirmatory Question. Prior to data analysis, Abt added the confirmatory 
research question “Were BCO participants more likely to obtain or advance in a job after program entry 
relative to a comparison group who received only FOC services?” based on our re-examination of the 
BCO logic model concerning participants’ outcomes from employment services. 


Exploratory Employment Questions Not Addressed. Abt could not address the following two 
employment exploratory questions due to the difficulty in gathering point-in-time information noted 
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above: (1) Are BCO participants more likely to retain the same job (for 180 days) relative to a comparison 
group who receive only FOC services? and (2) Are BCO participants likely to have been employed more 
quickly relative to a comparison group who receive only FOC services? 


Financial Well-Being Research Questions 
Approach to Research Questions. The intended approach and change in timeframe described for 


employment questions also applies to the financial well-being questions. 


Addition of Financial Well-Being Confirmatory Question. Prior to data analysis, Abt added the 
confirmatory research question “Were BCO participants more likely to have obtained or increased their 
credit scores after program entry relative to a comparison group who received only FOC services? based 
on our re-examination of the BCO logic model concerning participants’ outcomes from financial services. 


Addition of Financial Well-Being Exploratory Question. The following exploratory research question 
was added because of the emphasis on net-worth in financial coaching services: Were BCO participants 
more likely to have increased their net worth after program entry relative to a comparison group who 
received only FOC services? 


Exploratory Financial Well-Being Questions Not Addressed. Abt could not address the following 
exploratory financial well-being question due to a lack of data on on-time account payments: Are BCO 
participants more likely to have a greater percentage of on-time account payments 12 months after 
program entry relative to a comparison group who receive only FOC services? 
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Implementation Study 


The BCO implementation study examined the fidelity of the implementation of the BCO logic model 
(Appendix A, Exhibit A-1). The implementation research questions, shown in Exhibit 2, address the key 
activities associated with each of the four BCO components: contextualized bridge services, financial 
coaching and education, income support services, and employment services. 


Abt selected seven BCO programs as treatment sites, which are listed in Exhibit 4 (see Appendix B for a 
detailed description of the site selection process). The implementation study assessed the use of the BCO 
model in those sites. 


Exhibit 4. BCO Implementation Sites 


Designation Used 


Site Location in Report 
Brighton Center, Inc. Greater Cincinnati and Northern Kentucky A 
Chinese Community Center, Inc. Houston, TX B 
District 1199C Training and Upgrading Fund Philadelphia, PA C 
Instituto del Progreso Latino Chicago, IL D 
International Institute of Minnesota St. Paul, MN E 
Project for Pride in Living, Inc. Minneapolis, MN F 
Wesley Community Center, Inc. Houston, TX G 


This section of the report describes the design, measures, and data collection for the implementation study 
and the results from the fidelity study. 


Implementation Study Design, Methods, and Analysis 


Overview of Design 

Abt’s approach to the implementation study was to assess the extent to which treatment sites’ delivery of 
the BCO model adhered to the characteristics of the model as defined by LISC, and to understand how the 
implementation evolved over time depending on challenges that the sites experienced in implementing the 
model. This approach involved having a well-defined set of BCO components of services and associated 
activities that include features of the structure and processes used in delivering the intervention 
(Mowbray, Holter, Teague, & Bybee, 2003; Bond, Williams, Evans, Salyers, Kim, Sharpe, & Leff, 2000), 
and collecting data about the fidelity of implementation using multiple methods (Century, Rudnick, & 
Freeman, 2010). 


Measures 
The measures for BCO treatment sites’ fidelity of implementation of the BCO model are activities 


associated with each of the four BCO components—contextualized bridge services, financial coaching 
and education, income supports, and employment services (Exhibit 2). 


Abt worked with LISC to develop this model of BCO components and associated activities through a 
multi-step process. Abt staff prepared draft BCO core component activities based on our review of the (1) 
LISC/SIF Request for Proposal (September 14, 2015), (2) LISC staff meeting notes from September 23, 
2015, and (3) reporting forms for LISC’s Salesforce data system. Abt staff discussed the BCO component 
activities with LISC evaluation and BCO program staff (May 2016), and BCO program staff provided the 
Service Integration Retreat PowerPoint presentation (March 2016) as additional information for Abt to 
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use in refining the BCO component activities. Based on the results from discussions with LISC staff and 
their review of our draft component activities, Abt prepared a final list of BCO component activities (June 
2016). These activities are included in the BCO logic model (Appendix A). 


Data Collection 

Data to address the implementation study question To what extent did the treatment sites implement the 
BCO program with fidelity? were collected from the treatment sites using three methods: (1) interviews 
with evaluation sites’ staff as part of two rounds of site visits that Abt conducted; (2) observations of 
services delivered in the BCO treatment sites during the site visits; and (3) conduct of interim telephone 
conferences during the two-year period of sites’ implementation of BCO services. Prior to the beginning 
of data collection Abt sent each treatment site a “letter of participation” that described the data collection 
activities which Abt would conduct and the site’s role in the data collection. The letter discussed the 
treatment site’s role in working with Abt to organize each site visit, including identifying individuals to 
be interviewed and services to be observed, scheduling the interviews, providing private space for the 
conduct of the interviews, and arranging for Abt staffs observation of services. The letter also provided 
information about the purpose of and topics for the telephone conferences. 


Site Visits (Interviews, Observations) 

The site visits were a mechanism for collecting data about the treatment sites’ implementation of the four 
BCO components. Data points were actual services delivered, staff's perceptions of services and their 
utility for BCO participants, methods and materials used in the delivery of services, and treatment sites’ 
partners in BCO services. The site visits involved: 


e The conduct of face-to-face interviews with BCO staff and partners using a structured interview 
protocol; 


e Observations of delivery of bridge instruction (basic skills instruction and occupational training), 
and job readiness services held in a group setting; and 


e Review of materials used in the delivery of BCO services. 


Abt’s senior research staff trained in observation and interview methods conducted the site visits, which 
occurred during year | (2017-2018) and year 2 (2018-2019) of the treatment sites’ participation in the 
evaluation. The two site visits enabled Abt’s staff to gather information over a sufficient period to identify 
any implementation issues that sites experienced with the BCO model. 


Listed in Exhibit 5 are the implementation study’s constructs that were assessed, the topics for questions 
that defined each construct, and method of data collection. Abt developed an implementation study 
protocol that had questions for each topic listed under each construct in Exhibit 5. The protocol was used 
to guide the two site visits and the interim telephone conferences. Abt adapted a class observation form 
used in its prior adult education studies to document (1) activities that the instructor or site staff member 
conducted during each observation, (2) clients’ participation in the activities, (3) materials used, and (4) 
other related activities that took place. 
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Exhibit 5. List of Implementation Study Constructs Assessed, Question Topics Addressed, and 
Data Collection Methods 


Cea aXoye Mo) im BY-1e- Mexe) (seri Co) 


Review of 
|= S016) 
Application, —_ Interview Interim 
Screening duringSite Program | Telephone 
Constructs and Topics for Questions Defining Constructs Calls Visits? Observation |Conference* 
Goals for BCO Grant 
Goals for number of clients to be served during 2017 and 2018 X Site Visit 1 
only 
Progress in numbers of clients served in training programs X 
Target Population for BCO 
Population organization currently serves X Site Visit 1 
only 
Subpopulation targeted for BCO training programs X X 
Partnerships formed to support BCO services being delivered X X 
How the partnerships work in terms of activities, staff involved X 
Benefits to partners for participating X 
How organization's services are advertised X 
How clients say they find out about BCO program X 
Within-program recruitment for BCO X 
Outcomes advertised for clients X 
Intake and Orientation X 
Activities conducted during intake and format for activities X 
Orientation activities, length of time, format, and next steps X 
Types of services offered, topics, and timing of services X X 
Completion of Financial Planning forms, review of credit score X 
Providers of services and location X 
Use of financial products, which products, and how used X 
Financial coaching approach, timing, topics discussed, and staff X 
Income Supports X 
Types and details of income supportive services offered X X 
Method used for public benefits screening X 
Sequence and timing of services, mode of delivery, role of coach X 
Types and details of job readiness, employment search services X X X 
Sequence of services delivered, format, and length of services X 


ies) 


During Site Visit 1, all topics were addressed. During Site Visit 2, program staff were asked to provide updated information 
on services conducted since Site Visit 1. 


- 


The interim telephone discussions addressed the status of data collection in Salesforce and updates in services that occurred 
since the last communication with Abt. 
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(eda aXoye Mo) im BY-ie- Mexe) I(-ert Coy a) 


Review of 
BCO : 
Application, —_ Interview Interim 
Screening duringSite Program | Telephone 
Constructs and Topics for Questions Defining Constructs Calls Visits? Observation |Conference* 
Staff and partners who provide services X 
Employment coaching approach, timing, topics, and staff : 


Job placement activities conducted, relationship with employers 


Bridge Services: Contextualized Adult Education (AE) 


Basic skills assessments used, testing schedule, and tester 
Basic skills assessment criteria for program participation 


Types of AE services offered, service providers, and location X X 
Instructional content, approach, delivery modality, and schedule X 
Approach for integrating occupational information into basic X 


skills 

ype of eimieluin Gonlexialization i career, completion point 
Occupational ‘aining type and industry/growth 

Occupation training length, delivery mode, location, and provider 
| Education and skill requirements for participation | CX : 
Credeiiials Gamed and requinetnietiis for credentials 
Career planning activities content, format, and timeline 
Relationship saves academic activities and career pathways 
Services provided to clients after they complete Bridge Program 
How clients access follow-up services 


Data Collection and Entry into Salesforce 


Process and timelines for data collection and data entry 


Prior to each site visit, Abt’s team conducted a planning telephone call with each treatment site’s BCO 
program coordinator to discuss the topics that would be addressed during the interviews, types of 
individuals to interview who could address the topics, the service to observe, and possible dates for the 
site visit. After the planning call, Abt sent a draft site visit agenda that was scheduled over a two-day 
period and involved a senior and a mid-level Abt staff member. The BCO site staff prepared a final 
schedule that included individual, one-hour interviews with the two Abt staff and observation of services 
if an observation was to be conducted during that visit. The categories of individuals for the interviews, 
which represent staff and external people involved in the BCO program, were: 


e Organization’s executive director or senior staff member; 
e BCO program coordinator or manager; 
e Staff who conduct client recruitment, intake, and orientation activities; 


e Bridge service staff including adult education instructor, occupational training instructors or 
vendors, and career advisor; 
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e Financial coach and staff involved in delivering or arranging financial education activities; 

e Employment coach and staff who conduct job readiness activities and job placement activities; 
e Income support services staff; 

e BCO program partners and employers; and 


e Staffmember who works with LISC’s Salesforce database. 


Abt’s site visit team interviewed the individuals listed above during the two site visits except for the 
program partners, who were interviewed during only one visit for most of the sites. 


In preparation for Site Visit 1, Abt’s site visit team reviewed each treatment site’s BCO application and 
site screening interview notes to record information in the protocol so that this information could be 
confirmed or updated during the site visit. All topics in Exhibit 5 were addressed during Site Visit 1. 
During Site Visit 2, BCO site staff were asked whether BCO activities in each of the four BCO 
components had changed or whether there were new activities since Site Visit 1. Abt’s team conducted at 
least one observation during at least one site visit. The services observed were career planning activities, 
basic skills instruction, or occupational training. The timing of the observations depended on the schedule 
of site activities. The information collected during the site visits was stored in a qualitative database that 
Abt established for each treatment site. The qualitative database was organized according to the 
constructs listed in Exhibit 5. Listed in Exhibit 6 is the schedule for the site visits and the number of 
individuals interviewed during each site visit. 


Telephone Conferences 

Abt’s staff conducted interim telephone calls with the BCO coordinator or other key staff at each 
treatment site in between the site visits to collect information about implementation of BCO services, 
modifications to existing services, new services added to the BCO components, difficulties encountered 
in implementing the BCO model, and status of the sites’ participant data entry into Salesforce (See 
Exhibit 5). The information from these telephone calls was entered into each treatment site’s qualitative 
database. 


Exhibit 6. Site Visit Schedule and Number of People Interviewed 


Site Visit Number and Number of People 

Site Schedule Interviewed 
Brighton Center, Inc. #1 May 16-17, 2017 9 

#2 November 28-29, 2018 9 
Chinese Community Center, Inc. #1 May 10-11, 2017 7 

#2 October 23-24, 2018 9 
District 1199C Training and Upgrading Fund #1 May 1-2, 2017 14 

#2 December 4-5, 2018 10 
Instituto del Progreso Latino #1 November 28-29, 20175 10 


Change to SEP. The SEP specified that two site visits would be conducted to each of the treatment programs as part of the 
implementation study. Abt conducted two site visits to six of the seven programs. Due to a scheduling difficulty, Abt could 
not conduct a site visit to the Instituto del Progreso Latino program but conducted telephone interviews with program staff 
to gather the same information that was to be collected during the second site visit. 
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Site Visit Number and Number of People 

Site Schedule Interviewed 
International Institute of Minnesota #1 September 21-22, 2017 11 

#2 June 5-6, 2019 10 
Project for Pride in Living, Inc. #1 July 13-14, 2017 11 

#2 October 29-30, 2018 11 
Wesley Community Center, Inc. #1 June 21-22, 2017 9 

#2 February 12-13, 2019 9 


Data Analysis 

The data collected through the site visits (interviews, observations) and interim telephone conferences 
were analyzed to assess the overall fidelity of delivery of the BCO model. A content analysis was 
performed to understand the types of activities that the treatment sites conducted for each of the BCO 
components. The content analysis involved organizing the information from the site visits and telephone 
conferences that was in each BCO site’s qualitative database in a separate analysis matrix for each site. 
The left column in each site’s matrix had a row for each of the BCO components and activities that are 
shown in Exhibit 2. The top row of the matrix had multiple columns for each of the two site visits and the 
interim telephone conferences. Abt’s site visit team prepared a matrix for each BCO site that listed the 
activities the site conducted for each BCO component by the data collection period in which the 
information was collected (i.e., Site Visit 1, Site Visit, telephone conference). The resulting matrix for 
each site showed all activities that Abt’s team had documented for that site during the period of the 
implementation study. 


Fidelity Analysis 

The activities listed in each BCO site’s matrix were analyzed to determine whether a treatment site had 
adhered to the BCO model, defined as whether the four components of the BCO model were delivered as 
intended. Abt developed a three-point scale to rate the degree to which each of the activities for each 
component was implemented as specified in the BCO list of components (Exhibit 2). 


e Rating of 1: implemented as intended; activity carried out as stated in the BCO model; 
e Rating of 0.5: partially implemented; part of the activity carried out; and 


e Rating of 0: not implemented; activity not carried out. 


Abt’s senior researcher used the adherence scale to rate the activities listed in each BCO site’s matrix in 
terms of whether the activities in each matrix aligned with the required activities for each BCO 
component as shown in Exhibit 2. Each site had to have at least one instance of alignment to each activity 
for each component to receive a rating of “1.” If a site implemented only part of the required activity, a 
rating of “0.5” was made. A second researcher then made an independent rating to validate the scale and 
the ratings. 


Findings from Fidelity Analysis 
Presented in Exhibit 7 are the ratings for each of the BCO sites for each activity in each BCO component. 
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Exhibit 7. Fidelity Ratings by Site for Each BCO Component 


BCO Treatment Sites’ Fidelity Ratings 


BCO Components and Associated Activities SiteA SiteB SiteC SiteD SiteE SiteF SiteG 
1A. Bridge Services Component-Instruction 

Addresses career pathway in growth sector or industry 1 1 1 1 1 1 1 
Includes an academic pathway 1 1 1 1 1 1 
Teaches basic skills that are contextualized to a specific 1 0.5 1 1 1 0 1 
career path 

Is based on industry-specific curriculum 1 1 1 il 1 1 1 
Enables clients to attain industry-recognized credentials 1 1 1 1 1 0.5 1 
Has measurable endpoint or clearly defined completion 1 1 1 1 1 1 1 
criteria 


1B. Bridge Service Component-Assessment 


Uses criterion levels to determine client participation 


Tests pre- and post-test skills : ; ; : : ; 
Provides ongoing career coaching 1 0.5 1 fl 1 1 1 
Connects academic and career pathways 1 0.5 1 0.5 1 1 1 


a eee 100% | 85% | 100% | 95% | 100% | 75% | 100% 
g g (10) | (8.5) | (10) | (9.5) | (40) | (7.5) | (10) 


Financial Coaching and Education Component 


Assesses Clients’ financial health, combining a financial 
profile, budget, balance, and credit information 


Provides regular one-on-one interactions concerning 1 1 1 0.5 1 1 1 
clients’ financial status 
Reviews clients’ credit reports every six months ; ; ; ; - 


Uses financial products during coaching 


Income Support Services Component 


Screens for public benefits eligibility 


Identifies income supports to enable participants to 
remain in training 

Reassesses clients’ needs as their situations change 1 0.5 0.5 0.5 1 1 1 
Total Financial Coaching and Income Support 100% | 92% 92% 85% | 92% | 100% | 100% 
Services Rating (7) | (6.5) | (6.5) | (6) | (6.5) | (7) (7) 


Employment Services Component 


Assists clients in preparing job search materials and 
conducting job search 


Provides clients with coaching about job search, 1 1 1 1 1 1 1 
placement, retention, and advancement 
Supports clients in exploring careers 1 0.5 1 1 1 0.5 1 

3 : 100% | 83% | 100% | 100% | 100% | 83% | 100% 
Total Employment Services Rating (3) (2.5) (3) (3) (3) (2.5) (3) 
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Presented in Exhibits 8, 9, and 10 are the results from the analyses of adherence to the BCO model by 
each BCO treatment site. Exhibit 8 shows the percentage of sites that adhered to implementing each 
activity specified for the BCO bridge services. Four of the sites implemented 100 percent of the activities 
“as intended,” one site implemented 95 percent of the activities, one site implemented 85 percent of the 
activities, and one site implemented 75 percent of the activities. 


Exhibit 8. Extent of Treatment Sites’ Adherence to BCO Bridge Services Model 


100% 
80% 
60% 
40% 
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0% 


Site A Site B Site C Site D Site E Site F Site G 


@ Treatment Sites 


The activities that were not implemented in Site F concerned the delivery of basic skills instruction. At 
intake, that site assessed its BCO treatment group as not needing basic skills enhancement and thus did 
not provide basic skills instruction. 


Exhibit 9 shows the percentage of sites that adhered to implementing each activity specified for the BCO 
employment services. Five of the sites implemented 100 percent of the activities “as intended.” The other 
two sites partially implemented one of the activities. 


Exhibit 9. Extent of Treatment Sites’ Adherence to BCO Employment Services Model 
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Exhibit 10 shows the percentage of sites that adhered to implementing each activity specified for the BCO 
financial coaching and education, and income support services. Three of the sites implemented 100 
percent of the activities “‘as intended;” three sites partially implemented one activity; and one site partially 
implemented two activities. 


Exhibit 10. Extent of Treatment Sites’ Adherence to BCO Financial Coaching and Education, and 
Income Support Services Model 


100% 
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Overall, the seven BCO treatment sites demonstrated a high degree of adherence to the BCO model with 
two of the sites implementing 100 percent of the BCO activities in the four components of services “as 
intended.” Of the four components, the employment coaching services had the highest degree of 
adherence to the BCO model, with five sites implementing 100 percent of the employment services “as 
intended.” 
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Impact Study Findings 


The impact study examined whether the BCO model, which integrates bridge services with FOC services, 
better prepares participants with low incomes to increase their employment prospects and financial 
stability than do FOC services alone. To answer the research questions on employment and financial well- 
being, the Abt team used a quasi-experimental design and compared the outcomes of BCO participants 
(the treatment group) with a matched group of FOC participants (the comparison group). 


Across seven organizations’ BCO programs (Brighton Center, Inc.; Chinese Community Center, Inc.; 
District 1199C Training and Upgrading Fund; Instituto del Progreso Latino; International Institute of 
Minnesota; Project for Pride in Living, Inc.; and Wesley Community Center, Inc.) and six organizations’ 
FOC programs (Asociacion Puertorriquefios en Marcha, Cara, HumanKind, Prosperity Center for 
Financial Opportunity, Sacred Heart Community Service, and Urban League of Essex County), eligible 
BCO participants were exactly matched to one or more eligible FOC participant on gender, age, 
race/ethnicity, highest level of education completed, employment status, and credit score at intake using a 
coarsened exact matching (CEM) process°®. 


The final analytic sample that examined the confirmatory employment research question included 673 
treatment group participants (who received the BCO bridge services) and 566 comparison group 
participants (who did not receive the BCO bridge services); the sample for the analysis of the 
confirmatory financial well-being research question included 181 treatment group participants and 79 
comparison group participants (Exhibit 11). 


A detailed description of this impact evaluation’s design, sample recruitment and retention, baseline 
equivalence, methods, and analysis procedures are in Appendix B. 

Impact Study Questions, Data Sources, and Analytic Approach 

Confirmatory Research Questions 


This section of the report is organized by the confirmatory research questions of interest’: 


e Were BCO participants more likely to obtain or advance in a job after program entry relative to a 
comparison group who received only FOC services? (Employment) 


e Were BCO participants more likely to have obtained or increased their credit scores after program 
entry relative to a comparison group who received only FOC services? (Financial Well-Being) 


6 — Jacus, King, & Porro, 2012; Berta, Bossi, & Verzillo, 2017. 


The study examined the effects of the BCO model on multiple measures of employment and financial well-being, as well as 
on the pre-post change of an education measure for BCO participants. The confirmatory measures of the study focus on job 
improvement and credit score increase after program entry. The study also examined exploratory measures of employment 
(wage increase, hourly wage, and number of hours worked); three exploratory measures of financial well-being (credit score 
increase, overall debt, and net worth); and one exploratory measure of education (reading skills). Appendix C provides the 
results of the exploratory measures. 
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Exhibit 11: Flow from Eligible BOC and FOC Participants to Analytic Study Sample 


Participated in BCO services — Participated in FOC services only — 
Treatment Group Comparison Group 


INCLUDED IN MATCHED SAMPLE: INCLUDED IN MATCHED SAMPLE: 


INCLUDED IN ANALYTIC SAMPLES (NON-MISSING OUTCOME): 
Employment Outcome 


Financial Outcome 


Impact Study Data Sources 

Because study participants were drawn from each site’s existing service population, all analyses were 
based on extant administrative data that the sites submitted to LISC. BCO and FOC site staff collected 
and stored all client demographic and background information, service receipt, and outcome data in their 
Salesforce data system using the same procedures. Staff collect baseline data when clients enter a BCO or 
an FOC-only program, prior to the delivery of an intervention; document the activities in which clients 
participate as services are provided (within 48 hours of data receipt); and collect follow-up data on 
clients’ employment, financial well-being, and education outcomes (when clients initiative contact with 
staff or staff contact clients). 


The impact study job advancement data, for BCO and FOC-only participants were drawn from Salesforce 
advancement and employment records and the credit score data were drawn from the Salesforce credit 
score record. 


Analytic Approach 

Participant outcomes were analyzed in a linear regression model that tested the effect of BCO on job 
advancement and credit score increase. To account for the matching block design, we estimated the 
impact of BCO using models that include indicators for the matching blocks used in the design. Those 
indicators ensured that the overall estimate of the treatment-comparison group outcome difference was 
calculated by comparing the BCO and FOC participant outcomes within the matched blocks and 
calculating a precision-weighted average of the within-block differences to produce the overall estimate. 
To improve precision and control for possible baseline differences between the groups that might be 
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related to outcomes, the model also included covariates for participants’ gender, age, race/ethnicity and 
baseline level of education, employment, and credit score. 


For the reading test exploratory analysis, learners’ raw scores were converted into scale scores. Scores 
that represent the amount of a participant’s skill change from pre- to post-tests were constructed by 
subtracting each participant’s pre-test scale score from their post-test scale scores. The statistical 
significance of the mean gain was assessed using the paired t-test. 


Findings from Impact Analysis 


This section presents the results from the analyses of the impact study’s confirmatory research questions. 
Detailed information about the results from the confirmatory analyses are presented in Appendix C in 
Exhibit C-1 (Impacts on Employment) and Exhibit C-3 (Impacts on Financial Well-Being). 


BCO increased the share of participants who had a job improvement. 

The BCO model assumes that contextualized bridge services which are integrated with financial coaching 
and education, income support services, and employment services will benefit adults with lows skills or 
who have difficulty obtaining employment. BCO services are expected to move adults with low skills and 
low incomes toward family-sustaining employment by first increasing their basic skills and credentials, 
which in turn, can enable them to qualify for jobs with family-living wages. Hence, the study examined 
whether BCO participants were more likely to have a job improvement after program entry relative to a 
comparison group who received only FOC services. Overall, participants who were in the BCO programs 
were 19 percentage points more likely to have a “job improvement’’’ than were participants in FOC-only 
programs (56 percent versus 37 percent). These results are statistically significant at p<.001 (Exhibit 12). 


Exhibit 12. Impact of BCO on Whether Participants had a Job Improvement after Program 
Enrollment 


100 
90 
80 
70 
60 55.6 *** 


50 
36.8 


Percentage of participants 


m BCO Participants FOC Participants 


Note: ***Difference is statistically significant at the .001 level. Data include 673 participants in BCO programs and 566 participants in FOC 
programs. Percentage of participants represents those who had a “job improvement’ in 1-14 months after program enrollment. 
Source: BCO and FOC administrative data in Salesforce: FFT™ Employment Record/Job Advancement Record. 


Job advancement is defined as follows: the participant either (1) enrolls in the program unemployed and starts a new job in 
the 1-14 months after enrollment or (2) enrolls in program employed and has one of the following records in the 1-14 
months after enrollment: (a) obtains a new job and the new wage is greater than the wage at enrollment, or (2) has an 
advancement record for an increase in wages or a promotion. 
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There was No Significant Effect of BCO on Improving Participants’ Credit Score 

BCO services are intended to provide clients with financial coaching and education that will enable them, 
over time, to increase their assets and financial stability. The impact study examined whether BCO 
participants were more likely to have an increase in their credit score after program entry relative to a 
comparison group who received only FOC services. About 68 percent of participants in the BCO program 
and about 80 percent of participants in the FOC-only program had an increase in their credit score after 
program entry. The results were not statistically significant (Exhibit 13). 


Exhibit 13. Impact of BCO on Credit Score Increase after Program Enrollment 
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Note: Data include 181 participants in BCO programs and 79 participants in FOC-only programs. Percentage of participants represents those 
whose maximum credit score after program enrollment was higher than their credit score at intake. 
Source: BCO and FOC administrative data in Salesforce: FFT™ Credit Score Record. 


Descriptive Analyses of Treatment Group’s Education Outcomes 


Abt conducted descriptive analyses of the BCO treatment group participants’ education outcomes to 
understand participants’ progress in developing their foundational skills while enrolled in BCO, in 
completing BCO, and in earning educational credentials. The findings are discussed below. 


Information about each BCO treatment site’s TABE® results is presented in Appendix C, Exhibit C-5- 
(TABE® Pre-Post Test Results) and Exhibit C-6 (TABE Pre-Test Grade-Equivalent Levels). 


BCO Participants’ Foundational Literacy Skills 

As part of BCO’s bridge services, BCO treatment sites were to provide contextualized instruction to 
increase the foundational literacy skills of BCO participants who had low basic skills. To examine the 
effects of BCO on literacy skills, we compared the pre- and post-scale score on the Test for Adult Basic 
Education (TABE®) for all BCO participants (not just those in the matched samples). Of the 1,133 
treatment group participants in the evaluation, 594 individuals had TABE® Reading pre-test data, and of 
those with valid pre-test data, 282 (47.5%) had valid post-test data. 


Presented in Exhibit 14 are the results of the TABE® Reading pre-posttest analysis. BCO participants’ 
average TABE” pre-test score of 532 is equivalent to approximately the 6.6 grade-level. At the post test, 


Abt Associates BCO Evaluation Final Report April 2021 Jj 20 


IMPACT STUDY FINDINGS 


BCO participants had increased their reading scores by 9 points or by about one grade-level to the 7.4 
grade-level equivalent (CTB/McGraw-Hill, 2004). This increase was statistically significant at p<.001. 


Exhibit 14. TABE® Reading Pre-Post Results 


Average — Average Average SI ETIOET Ce! Standardized P-Value of 

Pre-Test  Post-Test Pre- to Post- Deviation of Test Score Gain from 
Outcome Score Score Test Gain Score Pre-Test Gain Paired t-Test 
TABE® Reading 532 541 9 65.99 0.14 0.0002” 
Scale Score 


Note: *** Difference is statistically significant at the .001 level. Data include 282 participants in BCO programs that had pre- and post-test 
TABE® Reading scores. 
Source: BCO administrative data in Salesforce: FFT™ TABE® record. 


Analysis of Treatment Group’s Completion of Education or Training 

We analyzed the extent to which the treatment group (1,133 participants) completed their education or 
training program and obtained an educational or industry credential by the time of follow-up. As shown in 
Exhibit 15, at follow-up, the majority (86.1%) of treatment group participants had completed their BCO 
education or training program. As shown in Exhibit 16, of the treatment group participants with data who 
completed training or education, the majority (79.7%) also earned either an education credential or an 
industry credential. In one BCO program, the program’s completion certificate was the endpoint for 
success in that program and was recognized by local employers. 


Exhibit 15. Education or Training Program Completion Status of Treatment Group 


Training Program Completion Status for Treatment Group | Frequency Percentage 
Completed education/training program 878 86.1% 
Did not complete education/ training program 142 13.9% 

N Missing: 113 


Source: BCO administrative data in Salesforce: FFT™ Education/Training Record. 


Exhibit 16. Number of Credentials Earned by Treatment Group 


Number Frequency Percentage 
1 586 79.7% 
2 134 18.2% 
3 14 1.9% 
4 1 0.1% 


N Missing: 398 
Source: BCO administrative data in Salesforce: FFT™ Education/Training Record. 


We analyzed the last credential that treatment group participants earned by the time of follow-up. As 
shown in Exhibit 17, more than half (56.9%) of treatment group participants earned a credential in a 
healthcare occupation, such as Certified Nurse Assistant and Medical Assistant, which reflects the type of 
occupational training offered in BCO programs. 
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Exhibit 17. Type of Credentials Earned by Treatment Group 


Credential Type | Frequency Percentage 
Healthcare 418 56.9% 
Child Development 64 8.7% 
NIMS -Manufacturing 42 5.7% 
Hospitality 35 4.8% 
Business Technology 23 3.1% 
Environmental 0 0.0% 
Educational degree 122 16.6% 
Ancillary certificates and licenses 31 4.2% 


N Missing: 398 


Source: BCO administrative data in Salesforce: FFT™ Education/Training Record. 
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Conclusion 


Conclusions 


The evaluation found that the BCO model, with well-defined services and training and technical 
assistance support, can be implemented by community-based organizations with strong adherence to the 
model. The evaluation also indicated that BCO’s goal to facilitate clients’ attainment of educational 
credentials and employment on their path to economic well-being can be attained through the delivery of 
the multi-component services that define the BCO model. Although BCO evaluation participants were not 
able to make significant advances in their economic status during the 14 months after they enrolled in the 
BCO program, they were able to earn credentials and advance in their jobs. Those outcomes are thought 
to be steps toward economic well-being. The process of increasing individuals’ economic status is 
complex and affected by many factors. Although this evaluation provided preliminary insights about near- 
term changes in participants’ economic status, more robust investigations with larger samples of 
participants are needed to understand adults’ attainment of economic well-being. 


Fidelity of BCO Implementation 

The seven treatment sites implemented the BCO model’s four components of activities with a high degree 
of fidelity. Many factors contributed to this result including the ongoing training and technical assistance 
that LISC staff provided to its grantees and the organizational resources of the seven programs that 
participated as treatment sites. Another factor is that the BCO model is specified with sufficient detail to 
guide programs’ main activities in each of the four BCO components but is not so prescriptive that 
programs cannot adjust services as they encounter challenges. This model design enabled the treatment 
programs to respond to the needs of their client populations while delivering the range of services that 
compose the BCO model. 


Education Outcomes 
The majority of BCO treatment participants completed the BCO program (86%) and most of those 


completers earned at least one credential, which was primarily an occupational certificate (80%). The 
treatment sites’ approaches to recruitment and orientation helped clients understand the requirements and 
commitment needed to succeed in the BCO program. The BCO bridge design also provided clients with 
an opportunity to master skills in the bridge component that could contribute to their success in the 
occupational training. The ongoing support provided by BCO coaches and instructors helped clients to 
persevere and keep on track as they worked to earn an educational credential. 


Employment Outcomes 
The evaluation results indicated that BCO treatment participants were significantly more likely to obtain 


or advance in a job after enrollment than FOC comparison participants. Factors contributing to this 
outcome were the comprehensive job readiness and job placement services that the BCO sites provided. 
As the BCO implementation study indicated, the treatment sites’ employment services were the most 
highly aligned of the four BCO components of services such that BCO sites provided many similar 
services. Another factor likely contributing to the BCO participants’ job placement and advancement was 
the partnerships that BCO sites formed with local employers. BCO staff reported that those partnerships 
not only facilitated participants’ job attainment but also help support participants as they moved along in 
their employment. 


Financial Well-Being Outcomes 
The evaluation found that FOC comparison group participants were more likely to increase their credit 
scores than BCO treatment group participants. Comparison group participants also were more likely than 
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treatment participants to decrease their debt, increase their net worth, and have a higher amount of debt 
decrease. Comparison group participants’ earlier participation in the workforce after program enrollment 
may have enabled those participants to engage in financial activities helped to increase their financial 
well-being. 


Lessons Learned and Limitations 


One lesson from the evaluation concerns processes to mitigate the loss of administrative data. Although 
LISC and Abt encouraged sites throughout the evaluation to follow established protocols for entering 
client data, more frequent monitoring of the BCO and FOC data in Salesforce might have resulted in less 
loss of data. 


Because of the evaluation’s quasi-experimental design, the BCO treatment programs participating in the 
evaluation were not selected to represent the full set of BCO programs but were purposively selected 
because of their planned services, client population, stability of BCO program implementation, and 
projected number of participants. The FOC comparison programs were purposively selected for the 
evaluation because their clients who entered services during the evaluation’s enrollment period had 
similar background characteristics as the BCO clients in the treatment group. The BCO and FOC 
programs were also in cities with similar labor markets. For those reasons, the evaluation’s results may 
not be applicable to clients with background characteristics different from the backgrounds of evaluation 
participants or because of possible differences in BCO program services. 


The goal of using coarsened exact matching (CEM) in the evaluation was to create treatment and 
comparison groups that were similar, before the start of clients’ participation in the BCO and FOC 
programs, on observed characteristics likely to be related to the education, employment, and financial 
outcomes of interest. Therefore, any differences in education, employment, and financial outcomes 
observed can be attributed to the BCO program (or unobserved differences) rather than to initial 
differences between the groups. The treatment and comparison groups’ analytic samples were shown to 
be equivalent at baseline on observed characteristics, even after the loss of sample due to missing 
outcome data. However, a limitation of the study design is that differences observed may be due to 
unobserved characteristics of participants or the BCO program. Further, because the missing data led to 
small sample sizes for the financial outcome, the evaluation may not be powered to detect small 
differences between the groups. 
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Appendix A: BCO Logic Model 


The Bridges to Career Opportunities (BCO) model assumes that contextualized bridge services which are 
integrated with financial coaching and education, income support services, and employment services will 
benefit adults with low skills and have difficulty finding employment. The BCO logic model is presented 
in Exhibit A-1. Inputs are the resources needed to ensure implementation of the BCO model. They 
include but are not limited to LISC technical assistance to BCO grantee sites to facilitate their 
implementation of the BCO model; funding to BCO grantees to help support the delivery of BCO 
services; and grantees that provide an organizational structure and community partnerships for operating 
BCO services. 


Those inputs are expected to enable organizations to carry out BCO activities. The BCO model has four 
program components: 


e A contextualized bridge program to increase clients’ basic skills within the context of an 
academic and career pathway in a specific growth sector or industry (e.g., healthcare); 


e Financial coaching and education to help clients solve specific financial problems; plan for 
financial stability; and connect to financial service providers and financial products for saving 
and building credit; 


e Income support services to help clients access public benefits for which they are eligible that 
can enable them to participate in and complete training; and 


e Employment services to provide job readiness training, job search, and job placement. 


Together those activities are hypothesized to change clients’ proximal outcomes of educational 
attainment, employment, and financial well-being. The distal outcomes, made possible by the proximal 
outcomes, reflect BCO’s goal of helping adults with low incomes improve their financial stability and 
obtain jobs with living wages. 
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LISC technical 
assistance 


Funding 


Community-based 
organizations 


Abt Associates 


Contextualized Bridge Program* 
Addresses career pathway in growth sector or industry 
Includes an academic pathway 
Teaches basic skills needed to access educational programs and 
that are contextualized to a specific career path, which in turn 
prepares clients for occupational skills training and “middle 
skills” jobs 
Based on industry-specific curriculum 
Enables clients to attain “industry-recognized” credentials 
Has measurable endpoint or clearly defined completion criteria 
Pre- and post-test skills assessments. Pre-test is used as part of 
clearly defined entry criteria 
Ongoing career coaching connects academic and career 
pathways 


Financial Coaching and Education 
Combined Financial Assessment 
Regular 1:1 interactions concerning financial status 
Credit reports run every 6 months 
Uses financial products during coaching 
Mav be integrated into bridge course 


Income Support Services 
Public benefits screening 
Supports to enable clients to remain in training 
Needs are reassessed as clients’ situations change 


Employment Services 
Assistance with preparing job-search materials and conducting 
job search 
Counseling about job search, placement, retention, and 
advancement 
Career exploration support 
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Education 

At program completion 
Increase basic skills 

12 months after program 

entry 


Improve financial stability 
and obtain family-sustaining 


employment 


Earn postsecondary or 
occupational credential 


Employment 

12 months after program 

entry 
Obtain job 
Increase hourly wages 
Obtain higher hourly 
wages in first placement 
Increase hours worked 
Increase job retention 
Decrease time to job 


Financial Well-Being 
12 months after program 
entry 
Obtain or increase credit 
score 
Increase percentage of 
accounts with on-time 
payments 


* Addition of the contextualized bridge program is the key 
difference between the FOC model and the BCO model. 


April 2021 | 26 


APPENDIX B: Additional Information about the Evaluation 
Design and Impact Study 


Appendix B: Additional Information about the Evaluation 
Design and Impact Study 


This appendix provides details about the evaluation design and its implementation to address the 
evaluation’s research questions. Discussed in this appendix are the research questions, sample, data 
sources, measures, and analytic methods of the evaluation. 


Research Questions 


The evaluation examined whether the BCO model, which integrates bridge services with FOC services, 
better prepared participants with low skills or who are hard to employ to increase their employment 
prospects and financial stability than FOC services did alone. The impact evaluation was guided by 
confirmatory and exploratory research questions. 


The confirmatory questions tested a key premise of the BCO model compared to the FOC model—that 
BCO programs, by providing education services to increase low-skilled clients’ foundational literacy and 
numeracy skills, can improve clients’ success in occupational training and credentialing programs and 
enable them to qualify for jobs with family-sustaining wages that lead to financial well-being. The Abt 
team examined two confirmatory research questions for the study in two separate domains: 


1. Were BCO participants more likely to obtain or advance in a job after program entry relative to a 
comparison group who received only FOC services? (Employment domain) 


2. Were BCO participants more likely to have obtained or increased their credit scores after program 
entry relative to a comparison group who received only FOC services? (Financial Well-Being 
domain) 


All other research questions were exploratory and concermed the effects of BCO services on outcomes in 
the domains of employment, financial well-being, and education. 


Exploratory Research Questions Regarding Employment 

Because BCO services are intended to place clients in better jobs by first increasing their basic skills and 
credentials, employment impacts are expected to be realized after clients complete occupational training. 
For the impact study, this period was 1-14 months after clients’ enrollment in BCO services.? 


Abt’s approach to the employment data analyses, as described in the Abt’s evaluation plan (Gan et al., 2016)? was to 
compare the treatment and comparison groups’ results for a number of employment outcomes at 12 months after 
participants’ enrollment in BCO services (treatment group) or FOC services (comparison group). The timeline assumed that 
BCO and FOC local staff would conduct follow-up data collection 12 months after enrollment. However, because the BCO 
model allows for client-driven interactions, only a portion of the follow-up data collection occurred at 12 months after 
enrollment. This client-driven approach made it difficult to gather point-in-time information, so the model itself did not 
necessarily lend itself to a 12-month follow-up. Rather, sites collected follow-up data, defined as any data collected after 
baseline, beginning in month | after enrollment and continued throughout the 14-month period in which Abt tracked clients’ 
participation in BCO or FOC services. BCO and FOC staff also reported that part of the variation in the timeline of follow- 
up data collection was due to sites’ difficulty in reaching clients after they ended their participation in program services. Abt 
analyzed employment outcomes beginning in month one after enrollment through month 14, which was the longest period 
all clients in the evaluation could participate in follow-up data collection. 
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The exploratory research questions to assess impacts on employment are: 


e Were BCO participants more likely to increase their hourly wages after program entry relative to 
a comparison group who received only FOC services? 


e Were BCO participants more likely to be placed in jobs with higher hourly wages relative to a 
comparison group who received only FOC services? 


e Were BCO participants more likely to work more hours per week after program entry relative to a 
comparison group who received only FOC services? 


Exploratory Questions Concerning Financial Well-Being 

BCO services are intended to provide clients with financial coaching and education that will enable them, 
over time, to increase their assets and financial stability. For the impact study, this period is 1-14 months 
after clients’ enrollment in BCO services." 


The exploratory research questions to assess impacts on financial well-being are: 


e Were BCO participants more likely to have lower overall debt after program entry relative to a 
comparison group who received only FOC services? 


e Were BCO participants more likely to have increased their net worth after program entry relative 
to a comparison group who received only FOC services?!! 


Exploratory Question Concerning Education 

The exploratory research question’? in the education domain provides context for the evaluation and was 
examined for the treatment group only, as the comparison group did not participate in a basic skills 
assessment as part of their FOC participation: 


e Were BCO participants likely to increase their reading skills (as measured by a standardized test) 
after participating in the BCO program? 


To answer the employment and financial well-being research questions, Abt used a quasi-experimental 
design and compared outcomes of BCO participants’ (the treatment group) to a matched group of FOC 


Abt’s approach to the financial well-being analyses as described in Abt’s evaluation plan (Gan et al., 2016) was to compare 
the treatment and comparison groups’ performance on three financial well-being outcomes at 12 months after participants’ 
enrollment in BCO services (treatment group) and FOC services (comparison group). Similar to the employment analyses, 
only a portion of the follow-up data collection on financial well-being occurred at 12 months; rather, sites collected follow- 
up data, defined as any data collected after baseline, beginning in month 1 after enrollment and continued throughout the 14- 
month period in which Abt tracked clients’ participation in BCO or FOC services. Because of the timeline of the follow-up 
data collection, Abt revised its approach and analyzed financial well-being outcomes beginning in month 1 after enrollment 
through month 14, which was the longest period all clients in the evaluation could participate in follow-up data collection. 


The third question specified in Abt’s evaluation plan (Gan et al., 2016) concerned percentage of on-time account payments. 
Abt could not address this question due to lack of data on on-time account payments. Thus we examined increase in net 
worth. 


The confirmatory research question for the evaluation’s education outcome was whether BCO participants were more likely 
to have attained an occupational certificate or postsecondary credential 12 months after program entry relative to a 
comparison group who received only FOC services. Abt’s analysis of education participation data for the comparison group 
participants revealed than only four participants had records of attaining occupational certificates or postsecondary 
credentials at follow-up. Thus we could not address this question in the evaluation. 
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participants (the comparison group). The pre- and post-outcomes of BCO participants (the treatment 
group only) used a paired pre-post analysis to descriptively address the research question on education. 


Feasibility Study to Select Study Sample 


This section describes how the Abt team identified the BCO and FOC programs for the study, how 
eligible participants from each program were identified, how the comparison group was formed to address 
the research questions, the samples included for each analysis, and the baseline equivalence of the groups. 


Treatment Sites and Participant Sample 
Seven organizations’ BCO programs (treatment group) were purposively selected for the study and 1,133 


of their clients were eligible to be included in 
the evaluation. 


Site selection was a multi-step screening process 
illustrated in Exhibit B-1. LISC staff began with 
32 BCO subgrantee programs that serve the 
target population for this evaluation— 
individuals who are unemployed or 
underemployed (adults with low incomes and 
low skills) and have not experienced success in 
traditional education. Of those, they identified 
20 BCO grantee programs that were deemed to 
be high-capacity sites with prior experience in 
delivering the bridge component and had 
sufficient projected baseline enrollment to meet 
the requirements of the evaluation. 


Abt reviewed the 20 programs’ BCO 
applications and eliminated six programs from 
consideration because their timeline for 
implementation of bridge services did not align 
with the evaluation timeline, the structure of 
their bridge services was not conducive to the 
evaluation, and/or their anticipated enrollment 
numbers were not sufficient to support the 
evaluation. 


We then conducted two rounds of telephone 
interviews with the 14 remaining candidate sites 
to clarify their planned services, client 
populations (spring 2016), stability of BCO 
program implementation, and projected number 
of participants (summer 2016). The multiple 
rounds of screening were necessary to assess the 
programs’ BCO activities compared to the BCO 
model as it was articulated for the purposes of 


Exhibit B-1. Site Selection Process 


32 BCO Programs 


LISC makes nominations for 
the evaluation 
(Nov 2015) 


20 Programs 


Abt reviews BCO applications 
(Late 2015) 


14 Programs 


Abt conducts telephone 
interviews (Spring/Summer 
2016) 


Abt conducts site visits and 
screens local FOC sites (Fall 
2016) 
(Fall 2016) 


Final sample = 7 Programs 


the evaluation, and to identify the availability of comparison groups. 
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Based on this information and sites’ interest in participating in the evaluation, seven BCO programs were 
selected as treatment sites in fall 2016. The seven treatment sites are shown in Exhibit B-2. Because the 
treatment sites provide a range of occupational training programs aimed at different client populations, 
some of which are not the target population for this evaluation, we worked with each treatment site to 
specify the appropriate BCO bridge training programs (i.e., programs offering bridge and FOC services) 
to include in the evaluation. 


Eligible participants from those sites were clients in the appropriate BCO bridge training programs who: 
e enrolled in BCO services during the period April 1, 2017 and through December 31, 2018; 
e enrolled in an eligible training course; and 


e had consented to have their BCO data shared at the time of BCO enrollment. 


Exhibit B-2. Seven BCO Treatment Sites 


Location Eligible Participants 

Brighton Center, Inc. Greater Cincinnati and Northern Kentucky 217 
Chinese Community Center, Inc. Houston, TX 117 
District 1199C Training and Upgrading Fund Philadelphia, PA 130 
Instituto del Progreso Latino Chicago, IL 146 
International Institute of Minnesota St. Paul, MN 354 
Project for Pride in Living, Inc. Minneapolis, MN 24 

Wesley Community Center, Inc. Houston, TX 145 
Total 1,133 


Comparison Sites and Participant Sample 

Six organizations’ FOC programs (comparison group) were purposively selected for the study and 1,217 
of their clients were eligible to be included in the evaluation. The six comparison sites are shown in 
Exhibit B-3. 


LISC recommended possible FOC comparison sites, and Abt collected information about seven FOCs to 
assess their appropriateness. We reviewed extant information about the FOCs and conducted telephone 
interviews with the FOC leadership in each site. During the interviews, we gathered information about the 
sites’ delivery of FOC services and confirmed that the FOCs were not providing education services to 
FOC clients. Abt also determined the sites’ willingness to serve as comparison sites. 


After the interviews, we conducted a preliminary analysis of the sites’ Salesforce data for clients who had 
enrolled in FOC services during the period April 1, 2017, through December 31, 2018, the dates for 
enrollment of BCO clients participating in the BCO evaluation. Those data analyses examined the extent 
to which the background characteristics of clients who enrolled in the FOCs during the evaluation 
enrollment period were like the background characteristics of the treatment group. We also examined the 
comparability of the labor markets in the cities where the BCO and FOC sites were located. Based on 
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those analyses and review, Abt determined that the seven FOCs'? were suitable comparison sites because 
they: * 


e did not provide educational services for clients; 


e were in cities with similar labor markets and served FOC clients with similar demographic 
characteristics as the BCO sites selected for the study; and 


e collected the evaluation’s required data for FOC participants and delivered FOC services 
according to the FOC model. 


Exhibit B-3. Six FOC Comparison Sites 


Site Location Eligible Participants 
Asociacion Puertorriquefios en Marcha Philadelphia, PA 7 

Cara Chicago, IL 503 
cence Community Development Richmond, VA 53 
Prosperity Center for Financial Opportunity Kansas City, MO 62 

Sacred Heart Community Service San Jose, CA 351 

Urban League of Essex County Newark, NJ 241 

Total 1,217 


Eligible participants from these sites are clients in the appropriate FOC programs who: 


e enrolled in an FOC program (with no BCO services) during the period April 1, 2017 and 
December 31, 2018; and 


e had consented to have their data shared at the time of FOC enrollment. 


Exhibit B-4 shows the demographic characteristics of those BCO and FOC participants at program intake. 


Seven FOCs initially agreed to participate in the evaluation. However, after Abt began to work with the comparison sites 
regarding their follow up on missing client data, one FOC determined it did not have the staff to obtain missing data and 
decided to discontinue its participation in the evaluation. Thus six FOCs participated. 


Abt considered three options for selecting FOC programs that could serve as the comparison sites for the evaluation. One 
option was to use the FOCs in the same organizations selected as BCO treatment sites. We found that to meet their BCO 
enrollment goals, the treatment sites recruited BCO clients from their organization’s FOC programs, which meant the 
remaining FOC clients were not available for or not interested in participating in the bridge services at that time. They were 
not an appropriate comparison group because their motivation differed from that of the treatment group. Abt also considered 
other FOC programs in the same cities as the BCO treatment sites, which would control for labor market conditions. 
However, our review of client data in those FOC programs indicated that clients were not similar in demographic 
characteristics to the BCO participants eligible for the study. 
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Exhibit B-4. Characteristics of Eligible BCO and FOC Participants 


13009 =i fel]e) (24 FOC Eligible» 
Participants Participants 

2) ed) 
Female 
Race/Ethnicity 
Black/African American non-Hispanic 51.3 58.5 
White non-Hispanic 13.5 9.9 
Hispanic 24.1 25.1 
Other (American Indian, Asian, Multiracial, Biracial, Other) 1134 6.6 
17-24 27.8 13.8 
25-34 33.9 23.7 
35-44 19.4 18.6 
45-54 13.1 22.4 
55+ 5.9 21.5 
Divorced TA 8.9 
Married 24.6 12.7 
Separated 75 6.4 
Single 59.6 70.6 
Widowed 1.0 1.5 
English 62.0 86.3 
No High School Diploma/Equivalency 10.1 15.9 
High School Diploma or High School Equivalency 49.8 48.6 
Some College 23.9 14.6 
College Certificate or Degree 15.9 20.8 
Yes 40.9 20.3 
Exceptional (800+) 0.3 1.1 
Very Good (740-799) 7.5 3.6 
Good (670-739) 16.5 8.0 
Fair (580-669) 21.1 16.3 
Poor (<580) 29.6 25.0 
Insufficient Credit Score 23.5 40.8 
Non-FICO Credit Score Available 1.5 5.3 
Total 1,133 1,217 


4 Eligible participants are those who enroll in BCO services during the period April 1, 2017 and December 31, 2018, enrolled in an eligible 
training course and consented to participate in this evaluation. 

> Eligible participants are those who enrolled in a FOC program (with no BCO services) during the period April 1, 2017 and December 31, 2018 
and consented to participate in this evaluation. 

Sample Sizes: 

Gender: BCO Participants = 1,125 clients, FOC Participants = 1,203 clients 
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Race/Ethnicity: BCO Participants = 1,120 clients, FOC Participants = 1,207 clients 

Age at Intake: BCO Participants = 1,119 clients, FOC Participants =1,204 clients 

Marital Status at Intake: BCO Participants =1,099 clients, FOC=1,181 clients 

Primary Language Spoken: BCO Participants =1,085 clients, FOC Participants =1,179 clients 

Level of Education at Intake: BCO Participants =1,105 clients, FOC Participants =1,196 clients 

Employed at Intake: BCO Participants =1,108 clients, FOC Participants =1,160 clients 

FICO Credit Score Range at Intake: BCO Participants =720 clients, FOC Participants =816 clients 

Sources: BCO and FOC administrative data in Salesforce: FFT™ Intake Record, FFT ™ Employment Record, FFT™Credit Report Record. 


Formation of the Treatment and Comparison Group 

To ensure that the BCO and FOC participants were comparable at program entry for the impact analyses, 
a coarsened exact matching (CEM)'* '° was implemented. CEM controls for selection bias and other 
biases related to group assignment (although not perfectly) by accounting for multiple characteristics of 
individuals. With this method, the eligible BCO participants are exactly matched to one or more of the 
eligible FOC participants on the variables'’ shown in Exhibit B-5 across sites. 


Exhibit B-5. Characteristics of Eligible BCO and FOC Participants 


Domain Baseline Measures Used for Matching 


Demographic characteristic e Gender 
e Race/ethnicity 
e Age 
Education e Highest level of education completed 
Employment e Employment status 
Financial well-being e Credit score 


The full set of baseline measures is described in more detail in Exhibit B-10. Each of the matching 
variables was measured at program intake prior to the start of clients’ services (except for credit score, 


Change to SEP: The SEP specified that the impact evaluation would o use Propensity Score Analysis (PSA), a matching 
technique wherein participants in a program are matched to non-participants based on their participation propensity score. 
This technique uses characteristics measured before the start of a program to determine the probability (the propensity score) 
that participants are in the treated group. After assigning propensity scores to individuals, participants are placed into blocks 
(or matching strata) such that the treatment and control individuals within each block have approximately equal propensity 
to be in the treated group. PSA is useful for selecting comparison members on a large number (20 or more) of 
characteristics. Given the small number of variables identified for matching in the BCO evaluation, Abt selected the CEM 
approach instead of a PSA matching approach. Because all matching variables are combined into a summary score, PSA 
does not guarantee close or exact matches on variables of interest. However, CEM ensures exact matches on variables of 
interest (because treatment and comparison matches within the same block have identical values on coarsened matching 
variables). A drawback of CEM is that it does not work well with more than a few matching variables. 


16 Tacus, King, & Porro (2012); Berta, Bossi, & Verzillo (2017). 


Some of the matching variables were selected because they are baseline measures of the outcomes for the study—i.e., level 
of education, employment status, and credit score range. We also considered other variables that prior studies have shown to 
be related to or predictive of outcomes—that is, gender, age, race/ethnicity, marital status, type of training program, and 
income. Prior studies show that there is a correlation between types of training (e.g., for traditionally female jobs such as 
certified nursing assistant and child development associate) and income (traditionally female jobs pay less than entry-level 
traditionally male jobs), hence we included gender as a matching variable to address these issues. More than half of the 
participants were White non-Hispanic or Hispanic across treatment sites, so we recruited comparison sites with a majority of 
Hispanic or White non-Hispanic participants and matched on race/ethnicity. Marital status is important in that it is related to 
financial status, but the Abt team already matched on employment status and credit score, hence we did not match on marital 
status. We also could not match on income because there is a very high rate of missing data for all participants, so we used 
baseline employment status as a proxy for income. 
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which may be measured up to 5 weeks!® after intake). The variables identified for matching are the 
baseline measures of the outcomes of interest or have been found to be associated with (or predictive of— 
depending on the design) the outcomes of interest in previous studies. Matching on these variables should 
significantly account for observable differences between BCO and FOC participants (e.g., Roder 2016, 
Rankin 2015). However, one way in which BCO and FOC participants may vary is in why they chose to 
enroll in the BCO or FOC program, which the evaluation was unable to observe. 


The CEM was applied to the data using a SAS macro written by Berta, Bossi, & Verzillo (2017). In this 
process, blocks were created where within blocks’ participants were exactly matched on highest level of 
education, employment status, gender, race/ethnicity, age, and credit score. Blocks were created by 
forming categories for the BCO participants across the matching variables (e.g., an employed, white 
female, 25-34 years old with a high school diploma and a credit score between 500 and 550) and finding 
FOC participants with the exact same combination of characteristics. If a given block did not contain at 
least one BCO and one FOC participant, it was dropped. 


Abt formed separate comparison groups for the employment and financial impact analyses because of the 
high percent of missing values on credit scores for both the BCO and FOC participants. There is no 
correlation between credit score and employment outcomes. Hence, credit score was included in the 
match when looking at financial outcomes but was not included in the match when examining educational 
outcomes. Both matches only included participants with non-missing data on all matching variables. 
Exhibits B-6 and B-7 list the number of participants matched across sites for both comparison groups. 


The BCO and FOC sites’ downloading of clients’ baseline credit scores generally occurred during the first meeting between 
clients and their financial coach. For BCO clients whose bridge training program did not begin at time of program 
enrollment because of the schedule, the meeting between clients and their financial coach may not occur until after clients 
began the training, which could be about five weeks after enrollment. 
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Exhibit B-6. Employment Outcomes: Number of Participants Matched per Site 
(excludes Credit Score from the Match) 


Site Number of Matched BCO Participants 
BCO Sites 
Brighton Center, Inc. 182 
Chinese Community Center, Inc. 102 
District 1199C Training and Upgrading Fund 112 
Instituto del Progreso Latino 131 
International Institute of Minnesota 303 
Project for Pride in Living, Inc. 22 
Wesley Community Center, Inc. 141 
Total 993 
Site Number of Matched FOC Participants 
FOC Sites 
Asociacion Puertorriquefios en Marcha 7 
Cara 448 
Humankind 49 
Prosperity Center for Financial Opportunity 51 
Sacred Heart Community Service 250 
Urban League of Essex County 170 
Total 975 


Exhibit B-7. Financial Well-Being Outcomes: Number of Participants Matched per Site 
(includes Credit Score in the Match) 


Site Number of Matched Participants 
BCO Sites 
Brighton Center, Inc. 26 
Chinese Community Center, Inc. 42 
Wesley Community Center, Inc. 76 
District 1199C Training and Upgrading Fund 33 
Instituto del Progreso Latino 29 
International Institute of Minnesota 99 
Project for Pride in Living, Inc. 2 
Total BCO 307 
Site Number of Matched Participants 
FOC Sites 
Asociacion Puertorriquefios en Marcha 6 
Cara 227 
Humankind 22 
Prosperity Center for Financial Opportunity 20 
Sacred Heart Community Service 72 
Urban League of Essex County 10 
Total FOC 357 
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Sample Loss and Attrition 

Exhibit B-8 provides a detailed consort diagram that shows the number of eligible treatment and 
comparison participants, the number of participants included in the matched samples and the number of 
participants in the analytic sample for each outcome. The difference between the numbers in the matched 
sample and the number of participants included in the analytic sample for each outcome is the number of 
participants “lost to follow-up.” For example, for the Job Status Improvement outcome, the matched 
sample included n=993 T group members and n=975 C group members, while the analytic sample (which 
included only individuals with non-missing outcome data) included n=673 T group members and n=566 
C group members. In the T group 993-673 = 320 were lost because they had a missing outcome measure, 
while in the C group 975-566=409 were lost because they had a missing outcome measure. Because this 
evaluation uses all administrative data and only includes non-missing outcomes in the analysis, there is a 
different amount of missing data for each outcome, so the “lost to follow-up” varies by outcome. 


The attrition rate for each outcome can also be calculated from Exhibit B-8. For example, for the Job 
Status Improvement outcome, the matched sample included n=993 T group members and n=975 C group 
members, while the analytic sample (which included only individuals with non-missing outcome data) 
included n=673 T group members and n=566 C group members. In the T group 993-673 = 320, (32%) 
were lost because they had a missing outcome measure. In the C group 975-566=409, (42%) were lost 
because they had a missing outcome measure. The overall attrition rate is 37% (320+409) / (993+975) 
and the differential attrition is 10% (42%-32%). Although this represents considerable sample loss due to 
missing outcome data, the results in Exhibit B-10 show that n=674 T and n=566 C group members that 
remained in the analytic sample were well matched on baseline characteristics. That is, the treatment and 
comparison group individuals in the analytic sample were practically identical to each other at baseline in 
terms of their gender, race/ethnicity, age, level of education, and employment status (as shown by the 
standardized effect size difference of zero between groups for each characteristic). Exhibits B-9 to B-22 
show that baseline equivalence is established for each analytic sample, as is required for a quasi- 
experimental design (QED). Further, Exhibit B-11 and Exhibit B-20 for the job status improvement and 
credit score confirmatory outcomes, respectively, show that the treatment and comparison participants 
lost at follow-up because of missing outcome data are also equivalent. For example, for the Job Status 
Improvement outcome, the 320 treatment participants (993-673=320) and the 409 comparison participants 
(975-566) were practically identical to each other at baseline in terms of their gender, race/ethnicity, age, 
level of education, and employment status (as shown by the standardized effect size difference of zero 
between groups for each characteristic). 
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Exhibit B-8. Flow from Eligible BOC and FOC Participants to Analytic Study Samples 


Participated in BCO services — Participated in FOC services only — 
Treatment group Comparison group 


Included in TABE pre-post (Individual n=1,133) (Individual n=1,217) 


analysis: 


Valid Pre- and Post-test 
score 


(Individual n=282) INCLUDED IN MATCHED SAMPLE: INCLUDED IN MATCHED SAMPLE: 


Matched on demographics excluding credit score Matched on demographics excluding credit score 
(Individual n=993) (Individual n=975) 


Matched on demographics including credit score Matched on demographics including credit score 
(Individual n=307) (Individual n=357) 


Takes Le Col-Ye Mlame-lar-ladrem-t-l18) ol(2.s 


Employment Outcomes 
Wage at Follow- Value of Hours Hours Worked 


Job 


Job Status Advancement Wage Increase Value of Wage Up Hours Worked Worked at 
Improvement* Increase Increase Increase Follow-up 
(Treatment (Treatment (Treatment (Treatment (Treatment (Treatment (Treatment (Treatment 
n=673) n=673) n=407) n=407) n=523) n=367) n=367) n=484) 
(Comparison (Comparison (Comparison (Comparison (Comparison (Comparison (Comparison (Comparison 


n=566) n=566) n=288) n=288) n=310) n=269) n=269) n=291) 


Financial Outcomes 


Maximum Credit Score at Follow-Up* Elvanslal worn 


(Treatment n=181) 
(Comparison n=79) 


Latest Credit Score at Follow-Up 
(Treatment n=177) 
(Comparison n=79) 


(Treatment n=194) 
(Comparison n=81) 


*Outcome used in confirmatory analysis. 
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Method for Testing Baseline Equivalence 

Abt estimated the standardized differences between the matched treatment and comparison groups at 
baseline to determine their level of similarity on all characteristics used in the matching. By construction, 
the treatment and comparison groups will be exactly matched on the CEM matching variables (highest 
level of education, employment status, gender, age, race/ethnicity and credit score range). 


To estimate the differences between BCO treatment and FOC comparison group participants at baseline, a 
modified version of the impact model (described below) that similarly accounts for the matching blocks 
design by including a set of indicators for the CEM matching blocks was fitted. The result is the average 
difference between treatment and comparison participants within each matching block. The model has the 
structure of Equation 1. 


Eq (1) Xxi = Ay + 4T; + Ln=1 SomDm + Eis 
where: 
Xxi, 7;, and D,,are as described in the impact model. 


a, represents the difference between BCO and comparison group learners on the k* baseline 
characteristic. 


Separate models are fit to each of the k baseline characteristics, and the reported difference at baseline 
is the coefficient @, from the model. 


Standardized differences less than or equal to 0.05 standard deviations do not require a statistical 
adjustment to the analysis models; regression adjustments are made for differences that are between 0.05 
and 0.25 standard deviations; and baseline equivalence is not met for any standardized difference greater 
than 0.25 standard deviations. The Abt team verified that the matched groups were similar on the baseline 
characteristics used for matching; that is, all standardized differences less than or equal to 0.05 standard 
deviations. 


Exhibits B-9 to B-22 show tests for similarity between the treatment and comparison groups for the initial 
match and each analytic sample used in the evaluation. By construction, the treatment and comparison 
groups are exactly matched on the CEM matching variables. Further, Exhibits B-11 and B-20 show the 
similarity between the treatment and comparison groups’ members that were lost for the two confirmatory 
outcomes. 
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Exhibit B-9. Employment Outcomes Initial Match: Baseline Equivalence of Matched Sample 
(excludes Credit Score) 


BCO Participants FOC Participants Standardized Effect 

Baseline Characteristic Mean Mean Size Difference 
Gender 
Female 85.9 85.9 0.0 
Male 14.1 14.1 0.0 
Black/African American non-Hispanic 53.3 53.3 0.0 
White non-Hispanic 12.0 12.0 0.0 
Hispanic 25.1 25.1 0.0 
Other (American Indian, Asian, Multiracial, 9.7 9.7 0.0 
Biracial, Other) 
Age at Intake 
17-24 27.2 27.2 0.0 
25-34 33.7 33.7 0.0 
35-44 19.3 19.3 0.0 
45-54 13.5 13.5 0.0 
55+ 6.2 6.2 0.0 
No High School Diploma/Equivalency 6.7 6.7 0.0 
High School Diploma or High School 51.4 51.4 0.0 
Equivalency 
Any College 42.0 42.0 0.0 
Employed at Intake 
No 59.9 59.9 0.0 
Yes 40.1 40.1 0.0 

Total N | 993 975 
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Confirmatory Employment Outcome 


Exhibit B-10. Baseline Equivalence of Matched Sample for Job Status Improvement Outcome 


BCO Participants FOC Participants § Standardized Effect 


Baseline Characteristic Mean Mean Size Difference 
Gender 

Female 84.8 84.8 0.0 
Male 15.2 15.2 0.0 
Black/African American non-Hispanic 53.9 53.9 0.0 
White non-Hispanic 10.9 10.9 0.0 
Hispanic 27.3 27.3 0.0 
Other (American Indian, Asian, Multiracial, 79 79 0.0 
Biracial, Other) 

Age at Intake 

17-24 27.8 27.8 0.0 
25-34 30.9 30.9 0.0 
35-44 20.8 20.8 0.0 
45-54 14.1 14.1 0.0 
55+ 6.4 6.4 0.0 
No High School Diploma/Equivalency 5.2 5.2 0.0 
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High School Diploma or High School 52.9 52.9 0.0 

Equivalency 

Any College 41.9 41.9 0.0 

Employed at Intake 

No 59.6 59.6 0.0 

Yes 40.4 40.4 0.0 
Total N 673 566 
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Exhibit B-11. Baseline Equivalence of Matched Sample Lost for Job Status Improvement Outcome 


BCO Participants OL Om az-TatCel| oy- 1) 9 
Excluded from Excluded from 
PVE ETS PVE ETS 
Mean Mean Standardized Effect 
Baseline Characteristic (n=320) (n=409) Size Difference 
Gender 
Female 88.1 88.1 0.0 
Male 11.9 11.9 0.0 
Black/African American non-Hispanic 51.9 51.9 0.0 
White non-Hispanic 14.4 14.4 0.0 
Hispanic 20.3 20.3 0.0 
Other (American Indian, Asian, Multiracial, 13.4 13.4 0.0 
Biracial, Other) 
Age at Intake 
17-24 25.9 25.9 0.0 
25-34 39.7 39.7 0.0 
35-44 16.3 16.3 0.0 
45-54 12.2 12.2 0.0 
55+ 5.9 5.9 0.0 
No High School Diploma/Equivalency 9.7 9.7 0.0 
High School Diploma or High School Equivalency 48.1 48.1 0.0 
Any College 42.2 42.2 0.0 
Employed at Intake 
No 60.6 60.6 0.0 
Yes 39.4 39.4 0.0 
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Design and Impact Study 


Exploratory Employment Outcomes 


Exhibit B-12. Baseline Equivalence of Matched Sample for Hourly Wage Increase Outcome 


BCO Participants FOC Participants © Standardized Effect 
Baseline Characteristic Mean Mean Size Difference 
Gender 
Female 88.7 88.7 0.0 
Male 11.3 11.3 0.0 
Black/African-America non-Hispanic 55.5 55.5 0.0 
White non-Hispanic 10.8 10.8 0.0 
Hispanic 24.8 24.8 0.0 
Other (American Indian, Asian, Multiracial, Biracial, 8.9 8.9 0.0 
Other) 
Age at Intake 
17-24 26.8 26.8 0.0 
25-34 31.0 31.0 0.0 
35-44 22.9 22.9 0.0 
45-54 14.7 14.7 0.0 
55+ 47 47 0.0 
No High School Diploma/Equivalency 5.4 5.4 0.0 
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High School Diploma or High School Equivalency 52.6 52.6 0.0 

Any College 42.0 42.0 0.0 

Employed at Intake 

No 75.9 75.9 0.0 

Yes 24.1 24.1 0.0 
Total N 407 288 
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Exhibit B-13. Baseline Equivalence of Matched Sample for Amount of Wage Increase Outcome 


BCO Participants FOC Participants Standardized Effect 
Baseline Characteristic Mean Mean Size Difference 
Gender 
Female 88.7 88.7 0.0 
Male 11.3 11.3 0.0 


Race/Ethnicity 


Black/African American non-Hispanic 55.5 55.5 0.0 
White non-Hispanic 10.8 10.8 0.0 
Hispanic 24.8 24.8 0.0 
Other (American Indian, Asian, Multiracial, 8.9 8.9 0.0 
Biracial, Other) 
Age at Intake 
17-24 26.8 26.8 0.0 
25-34 31.0 31.0 0.0 
35-44 22.9 22.9 0.0 
45-54 14.7 14.7 0.0 
55+ 47 47 0.0 
No High School Diploma/Equivalency 5.4 5.4 0.0 
High School Diploma or High School 52.6 52.6 0.0 
Equivalency 
Any College 42.0 42.0 0.0 
Employed at Intake 
No 75.9 75.9 0.0 
Yes 24.1 24.1 0.0 
Total N 407 288 
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Exhibit B-14. Baseline Equivalence of Matched Sample for Wage at Follow-Up Outcome 


BCO Participants FOC Participants | Standardized Effect 
Baseline Characteristic Mean Mean Size Difference 
Gender 
Female 85.9 85.9 0.0 
Male 14.2 14.2 0.0 


Race/Ethnicity 


Black/African American non-Hispanic 57.4 57.4 0.0 
White non-Hispanic 9.9 9.9 0.0 
Hispanic 24.1 24.1 0.0 
Other (American Indian, Asian, Multiracial, 8.6 8.6 0.0 
Biracial, Other) 

Age at Intake 

17-24 27.7 27.7 0.0 
25-34 31.9 31.9 0.0 
35-44 20.8 20.8 0.0 
45-54 14.5 14.5 0.0 
55+ 5.0 5.0 0.0 


Level of Education at Intake 


No High School Diploma/Equivalency 5.2 5.2 0.0 
High School Diploma or High School 50.7 50.7 0.0 
Equivalency 

Any College 44.2 44.2 0.0 
Employed at Intake 

No 59.1 59.1 0.0 
Yes 40.9 40.9 0.0 

Total N 523 310 
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Exhibit B-15. Baseline Equivalence of Matched Sample for Increase in Hours Worked Outcome 


BCO Participants FOC Participants Standardized Effect 
Baseline Characteristic Mean Mean Size Difference 
Gender 
Female 88.0 88.0 0.0 
Male 12.0 12.0 0.0 


Race/Ethnicity 


Black/African American non-Hispanic 53.7 53.7 0.0 
White non-Hispanic 11.4 11.4 0.0 
Hispanic 25.3 25.3 0.0 
Other (American Indian, Asian, Multiracial, 9.5 9.5 0.0 
Biracial, Other) 

Age at Intake 

17-24 27.8 27.8 0.0 
25-34 28.9 28.9 0.0 
35-44 22.6 22.6 0.0 
45-54 15.5 15.5 0.0 
55+ 5.2 5.2 0.0 


Level of Education at Intake 


No High School Diploma/Equivalency 5.7 5.7 0.0 
High School Diploma or High School 50.7 50.7 0.0 
Equivalency 

Any College 43.6 43.6 0.0 
Employed at Intake 

No 83.9 83.9 0.0 
Yes 16.1 16.1 0.0 

Total N 367 269 
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Exhibit B-16. Baseline Equivalence of Matched Sample for Mean Hours Worked Increase Outcome 


BCO Participants FOC Participants § Standardized Effect 

Baseline Characteristic Mean Mean Size Difference 
Gender 

Female 88.0 88.0 0.0 

Male 12.0 12.0 0.0 
Black/African American non-Hispanic 53.7 53.7 0.0 

White Hispanic 11.4 11.4 0.0 
Hispanic 25.3 25.3 0.0 

Other (American Indian, Asian, Multiracial, 9.5 9.5 0.0 


Biracial, Other) 
Age at Intake 


17-24 27.8 27.8 0.0 
25-34 28.9 28.9 0.0 
35-44 22.6 22.6 0.0 
45-54 15.5 15.5 0.0 
55+ 5.2 5.2 0.0 
No High School Diploma/Equivalency 5.7 5.7 0.0 
High School Diploma or High School Equivalency 50.7 50.7 0.0 
Any College 43.6 43.6 0.0 
No 83.9 83.9 0.0 
Yes 16.1 16.1 0.0 
Total N 367 269 


Abt Associates 


BCO Evaluation Final Report 


April 2021 | 46 


APPENDIX B: Additional 


Information about the Evaluation 
Design and Impact Study 


Exhibit B-17. Baseline Equivalence of Matched Sample for Hours Worked at Follow-Up Outcome 


BCO Participants 
Mean 


Baseline Characteristic 
Gender 
Female 


FOC Participants 
Mean 


Standardized Effect 
Size Difference 


Male 
Race/Ethnicity 
Black/African American non-Hispanic 


Biracial, Other) 


White non-Hispanic 10.3 10.3 
Hispanic 246 246 0.0 
Other (American Indian, Asian, Multiracial, 9.1 9.1 0.0 


Age at Intake 


17-24 28.5 28.5 0.0 
25-34 30.4 30.4 0.0 
35-44 20.7 20.7 0.0 
45-54 15.1 15.1 0.0 
55+ 5.4 5.4 0.0 


Level of Education at Intake 


No High School Diploma/Equivalency 5.4 5.4 0.0 

High School Diploma or High School Equivalency 49.0 49.0 0.0 

Any College 45.7 45.7 0.0 

Employed at Intake 

No 63.6 63.6 0.0 

Yes 36.4 36.4 0.0 
Total N 484 291 
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Exhibit B-18. Financial Outcomes Initial Match: Baseline Equivalence of Matched Sample (includes 
Credit Score) 


BCO Participants FOC Participants § Standardized Effect 


Baseline Characteristic Mean Mean Size Difference 
Gender 
Female 89.3 89.3 0.0 
Male 10.8 10.8 0.0 
Black/African American non-Hispanic 67.4 67.4 0.0 
White non-Hispanic 5.2 5.2 0.0 
Hispanic 22.8 22.8 0.0 
Other (American Indian, Asian, Multiracial, 46 46 0.0 
Biracial, Other) 
17-24 17.3 17.3 0.0 
25-34 34.2 34.2 0.0 
35-44 23.8 23.8 0.0 
45-54 15.6 15.6 0.0 
55+ 9.1 9.1 0.0 
No High School Diploma/Equivalency 78 78 0.0 
High School Diploma or High School Equivalency 52.4 52.4 0.0 
Any College 39.7 39.7 0.0 
No 75.2 75.2 0.0 
Yes 24.8 24.8 0.0 
A2 (FICO: 800-825) 0.0 0.0 0.0 
B1 (FICO: 770-799) 1.3 1.3 0.0 
B2 (FICO: 740-769) 1.0 1.0 0.0 
C1 (FICO: 705-739) 3.3 3.3 0.0 
C2 (FICO: 670-704) 2.3 2.3 0.0 
D1 (FICO: 625-669) 10.1 10.1 0.0 
D2 (FICO: 580-624) 10.8 10.8 0.0 
E1 (FICO: 440-579) 37.1 37.1 0.0 
XE1 (non-FICO: 451-600) 0.3 0.3 0.0 
Z (Zero/Insufficient Credit History) 33.9 33.9 0.0 
Total N 307 357 
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Confirmatory Financial Well-Being Outcome 


Exhibit B-19. Baseline Equivalence of Matched Sample for Maximum Credit Score Outcome 


BCO Participants FOC Participants § Standardized Effect 


Baseline Characteristic Size Difference 
Gender 
Female 91.7 91.7 0.0 
Male 8.3 8.3 0.0 
Race/Ethnicity 
Black/African American non-Hispanic 68.5 68.5 0.0 
White non-Hispanic 5.5 5.5 0.0 
Hispanic 23.8 23.8 0.0 
Other (American Indian, Asian, Multiracial, Biracial, 2.2 2.2 0.0 
Other) 
17-24 9.4 9.4 0.0 
25-34 35.9 35.9 0.0 
35-44 28.7 28.7 0.0 
45-54 16.6 16.6 0.0 
55+ 9.4 9.4 0.0 
No High School Diploma/Equivalency 2.8 2.8 0.0 
High School Diploma or High School Equivalency 50.8 50.8 0.0 
Any College 46.4 46.4 0.0 
No 66.9 66.9 0.0 
Yes 33.2 33.2 0.0 
A2 (FICO: 800-825) 0.0 0.0 0.0 
B1 (FICO: 770-799) 1.7 1.7 0.0 
B2 (FICO: 740-769) 1.1 1.1 0.0 
C1 (FICO: 705-739) 5.5 5.5 0.0 
C2 (FICO: 670-704) 3.3 3.3 0.0 
D1 (FICO: 625-669) 14.9 14.9 0.0 
D2 (FICO: 580-624) 17.1 17.1 0.0 
E1 (FICO: 440-579) 55.8 55.8 0.0 
XE1 (non-FICO: 451-600) 0.6 0.6 0.0 
Z (Zero/Insufficient Credit History) 0.0 0.0 0.0 
Total N 181 79 
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Exhibit B-20. Baseline Equivalence of Matched Sample Lost for Maximum Credit Score 


BCO Participants FOC Participants 


Excluded from Excluded from 
PVE WETS PVE WETS 
Mean Mean Standardized Effect 
PER MELE Cute (n=126) (n=278) Size Difference 
Gender 
Female 85.7 85.7 0.0 
Male 14.3 14.3 0.0 
Race/Ethnicity 
Black/African American non-Hispanic 65.9 65.9 0.0 
White non-Hispanic 48 48 0.0 
Hispanic 21.4 21.4 0.0 
Other (American Indian, Asian, Multiracial, 79 79 0.0 
Biracial, Other) 
17-24 28.6 28.6 0.0 
25-34 31.8 31.8 0.0 
35-44 16.7 16.7 0.0 
45-54 14.3 14.3 0.0 
55+ 8.7 8.7 0.0 
No High School Diploma/Equivalency 15.1 15.1 0.0 
High School Diploma or High School Equivalency 54.8 54.8 0.0 
Any College 30.2 30.2 0.0 
No 87.3 87.3 0.0 
Yes 12.7 12.7 0.0 
A2 (FICO: 800-825) 0.0 0.0 0.0 
B1 (FICO: 770-799) 0.8 0.8 0.0 
B2 (FICO: 740-769) 0.8 0.8 0.0 
C1 (FICO: 705-739) 0.0 0.0 0.0 
C2 (FICO: 670-704) 0.8 0.8 0.0 
D1 (FICO: 625-669) 3.2 3.2 0.0 
D2 (FICO: 580-624) 1.6 1.6 0.0 
E1 (FICO: 440-579) 10.3 10.3 0.0 
XE1 (non-FICO: 451-600) 0.0 0.0 0.0 
Z (Zero/Insufficient Credit History) 82.5 82.5 0.0 
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Exploratory Financial Well-Being Outcomes 


Exhibit B-21. Baseline Equivalence of Matched Sample for Latest Credit Score Outcome 


BCO Participants FOC Participants Standardized Effect 


Baseline Characteristic Mean Size Difference 
Gender 
Female 91.5 91.5 0.0 
Male 8.5 8.5 0.0 
Race/Ethnicity 
Black/African American non-Hispanic 68.4 68.4 0.0 
White non-Hispanic 5.7 5.7 0.0 
Hispanic 23.7 23.7 0.0 
Other (American Indian, Asian, Multiracial, 2.3 2.3 0.0 
Biracial, Other) 
17-24 9.6 9.6 0.0 
25-34 36.2 36.2 0.0 
35-44 28.8 28.8 0.0 
45-54 16.4 16.4 0.0 
55+ 9.0 9.0 0.0 
No High School Diploma/Equivalency 2.8 2.8 0.0 
High School Diploma or High School 50.9 50.9 0.0 
Equivalency 
Any College 46.3 46.3 0.0 
No 67.2 67.2 0.0 
Yes 32.8 32.8 0.0 
A2 (FICO: 800-825) 0.0 0.0 0.0 
B1 (FICO: 770-799) 1.7 1.7 0.0 
B2 (FICO: 740-769) 1.1 1.1 0.0 
C1 (FICO: 705-739) 5.7 5.7 0.0 
C2 (FICO: 670-704) 3.4 3.4 0.0 
D1 (FICO: 625-669) 15.3 15.3 0.0 
D2 (FICO: 580-624) 17.0 17.0 0.0 
E1 (FICO: 440-579) 55.4 55.4 0.0 
XE1 (non-FICO: 451-600) 0.6 0.6 0.0 
Z (Zero/Insufficient Credit History) 0.0 0.0 0.0 
Total N 177 79 
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Exhibit B-22. Baseline Equivalence of Matched Sample for Financial Net-Worth Outcome 


BCO Participants FOC Participants | Standardized Effect 


Baseline Characteristic Mean Mean Size Difference 

Female 88.1 88.1 0.0 
Male 11.9 11.9 0.0 
Race/Ethnicity 

Black/African American non-Hispanic 70.6 70.6 0.0 
White non-Hispanic 6.2 6.2 0.0 
Hispanic 20.1 20.1 0.0 
Other (American Indian, Asian, Multiracial, 3.1 3.1 0.0 
Biracial, Other) 

17-24 13.4 13.4 0.0 
25-34 35.1 35.1 0.0 
35-44 25.8 25.8 0.0 
45-54 15.0 15.0 0.0 
55+ 10.8 10.8 0.0 
No High School Diploma/Equivalency 5.2 5.2 0.0 
High School Diploma or High School Equivalency 50.0 50.0 0.0 
Any College 44.9 44.9 0.0 
No 76.8 76.8 0.0 
Yes 23.2 23.2 0.0 
A2 (FICO: 800-825) 0.0 0.0 0.0 
B1 (FICO: 770-799) 1.6 1.6 0.0 
B2 (FICO: 740-769) 1.0 1.0 0.0 
C1 (FICO: 705-739) 41 41 0.0 
C2 (FICO: 670-704) 2.6 2.6 0.0 
D1 (FICO: 625-669) 13.4 13.4 0.0 
D2 (FICO: 580-624) 12.9 12.9 0.0 
E1 (FICO: 440-579) 42.3 42.3 0.0 
XE1 (non-FICO: 451-600) 0.0 0.0 0.0 
Z (Zero/Insufficient Credit History) 22.2 22.2 0.0 

Total N 194 81 
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Data Collection 


Because evaluation participants were drawn from each site’s existing service population, all analyses 
were based on extant administrative data that the sites submit to LISC. The impact study data for BCO 
treatment group participants FOC-only comparison group participants were drawn from Salesforce, the 
commercial platform that LISC uses to store its client data. BCO and FOC site staff collect and store all 
client demographic and background information, service receipt, and outcome data in the system using 
the same procedures. Staff collect baseline data when clients enter a BCO or FOC-only program, prior to 
the delivery of an intervention; document the activities in which clients participate as services are 
provided (within 48 hours of data receipt); and collect follow-up data on clients’ employment, financial 
well-being, and education outcomes (when clients initiate contact with staff or staff contact clients). Abt 
identified the key variables in Salesforce to be used in the evaluation, and LISC staff transferred de- 
identified client data for those variables to Abt using Huddle™, a secure web portal. 


Analytic Methods 


Abt created measures about participants’ employment, financial well-being, and education, and estimated 
the effectiveness of BCO on the employment, and financial well-being measures for all participants. This 
section describes the approach we used to examine the effectiveness of the BCO. Described first is the 
rationale for selecting the evaluation’s measures that were used to examine the effects of BCO, and how 
those and other evaluation measures were constructed. Measures were constructed to capture (1) 
employment; (2) financial well-being; and (3) education. Discussed next are the evaluation’s analytic 
methods that were used to estimate effects of the BCO and FOC-only services on participants’ 
employment, and financial well-being measures. The final section provides the approach to examine the 
pre-post change for BCO participants’ education measure. 


Study Measures 
The evaluation examined the effects of BCO on multiple measures of employment and financial well- 


being for treatment and comparison participants, as well as the pre-post change of an education measure 
for BCO treatment participants. The evaluation’s confirmatory employment measure was job 
advancement after program entry and is discussed in the main body of the report. The evaluation also 
examined two exploratory measures of employment—hourly wages and number of hours worked after 
program entry. The evaluation’s confirmatory measure of financial well-being was credit score increase 
and is discussed in the main body of the report. The evaluation also examined two exploratory measures 
of financial well-being—overall debt and net worth. One exploratory measure of education—level of 
reading skills—is discussed in the main body of the report. Presented in Exhibit B-23 through Exhibit B- 
27 are the data sources and construction of the measures for the employment, financial well-being, and 
education measures, and participant characteristics. Because the two confirmatory measures were 
observed over a period of months, Exhibit B-24 shows the distribution of the number of months that 
elapsed between enrollment and measurement of the outcome. 


Presented in Appendix C are the results of the exploratory analyses. 


Abt Associates BCO Evaluation Final Report April 2021 1 53 


APPENDIX B: Additional Information about the Evaluation Design and Impact Study 


Exhibit B-23. Employment Outcome Measures Examined: Data Source and Measure Construction 


Measure Data Source Definition/Coding 

Job Status Improvement BCO and FOC | 1= (a) Enrolls in the program unemployed and then starts a new job in the 1-14 months after enrollment or (b) employed at 
enrollment and has one of the following records in the 1-14 months after enrollment: (1) gets a new job and the new wage is 
greater than the wage at enrollment; or (2) has an advancement record for an increase in wages or obtaining a promotion 
O= (a) Enrolls in the program unemployed, has record of follow up with the program but no record of new employment in the 
1-14 months after enrollment or (b) employed at enrollment, has a record of follow-up, but has no record of advancement or 
record of a new job with a higher wage 

Maximum Wage Increase at BCO and FOC | 1 = Maximum wage at follow-up in the 1-14 months after enrollment is greater than the wage at enrollment (wage defined as 

Follow-Up $0 for those unemployed at the time of enrollment) 
0 = Maximum wage at follow-up in the 1-14 months after enrollment is less than or equal to the wage at enrollment (wage 
defined as $0 for those unemployed at the time of enrollment) 

Value of Maximum Wage Increase | BCO andFOC | = The difference between the maximum wage at follow-up in the 1-14 months after enrollment and the wage at enrollment 

at Follow-Up 

Maximum Wage at Follow-Up BCO and FOC | = The maximum wage at follow-up in the 1-14 months after enrollment 

Latest Wage Increase at Follow- BCO and FOC | 1 =Latest wage at follow-up in the 1-14 months after enrollment is greater than the wage at enrollment (wage defined as $0 

Up for those unemployed at the time of enrollment) 
0 = Latest wage at follow-up in the 1-14 months after enrollment is less than or equal to the wage at enrollment (wage defined 
as $0 for those unemployed at the time of enrollment) 

Value of Latest Wage Increase at | BCO andFOC | = The difference between the latest wage at follow-up in the 1-14 months after enrollment and the wage at enrollment 

Follow-Up 

Latest Wage at Follow-Up BCO and FOC | = The latest wage at follow-up in the 1-14 months after enrollment 

Hours Worked Increase BCO and FOC | 1=Latest number of hours worked at follow-up in the 1-14 months after enrollment are greater than the number of hours 
worked at enrollment (hours worked defined as 0 hours for those unemployed at the time of enrollment) 
0 = Latest number of hours worked at follow-up in the 1-14 months after enrollment are less than or equal to the number of 
hours worked at enrollment (hours worked defined as 0 hours for those unemployed at the time of enrollment) 

Value of Hours Worked Increase BCO and FOC | = The difference between the latest hours worked at follow-up in the 1-14 months after enrollment and the hours worked at 
enrollment 

Hours Worked BCO and FOC | = The latest hours worked at follow up in the 1-14 months after enrollment 


Source: BCO and FOC administrative data in Salesforce. 
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Exhibit B-24. Number of Months Between Enrollment and Measurement of Confirmatory Outcomes 


Number of Months Between Enrollment and BCO Participants FOC Participants 
Measurement of Outcome ae) Ne) 
Job Status Improvement (Confirmatory) 

1 month 33 (4.9) 130 (23.0) 
2 months 48 (7.1) 126 (22.3) 
3 months 68 (10.1) 65 (11.5) 
4 months 94 (14.0) 44 (7.8) 
5 months 95 (14.1) 42 (7.4) 
6 months 61 (9.1) 24 (4.2) 
7 months 49 (7.3) 22 (3.9) 
8 months 31 (4.6) 25 (4.4) 
9 months 34 (5.1) 19 (3.4) 
10 months 28 (4.2) 15 (2.7) 
11 months 30 (4.5) 8 (1.4) 
12 months 29 (4.3) 8 (1.4) 
13 months 37 (5.5) 18 (3.2) 
14 months 36 (5.4) 20 (3.5) 
Credit Score (Maximum Credit Score at Follow-u 

1 month 0 (0.0) 0 (0.0) 
2 months 0 (0.0) 4 (5.1) 
3 months 3 (1.7) 6 (7.6) 
4 months 1 (0.6) 1 (1.3) 
5 months 3 (1.7) 2 (2.5) 
6 months 13 (7.2) 4 (5.1) 
7 months 14 (7.7) 4 (5.1) 
8 months 15 (8.3) 5 (6.3) 
9 months 10 (5.5) 2 (2.5) 
10 months 9 (5.0) 5 (6.3) 
11 months 7 (3.9) 2 (2.5) 
12 months 5 (2.8) 5 (6.3) 
13 months 2 (1.1) 5 (6.3) 
14 months 6 (3.3) 9 (11.4) 
15 months 9 (5.0) 2 (2.5) 
16 months 12 (6.6) 1 (1.3) 
17 months 9 (5.0) 3 (3.8) 
18 months 5 (2.8) 4 (5.1) 
19 months 3 (1.7) 4 (5.1) 
20 months T (3.9) 5 (6.3) 
21 months 6 (3.3) 1 (1.3) 
22 months 5 (2.8) 1 (1.3) 
23 months 8 (4.4) 0 (0.0) 
24 months 11 (6.1) 1 (1.3) 
25 months 2 (1.1) 1 (1.3) 
26 months 4 (2.2) 1 (1.3) 
27 months 1 (0.6) 0 (0.0) 
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Number of Months Between Enrollment and BCO Participants FOC Participants 
Measurement of Outcome Ne) 
28 months 4 (2.2) 0 (0.0) 
29 months 3 (1.7) 0 (0.0) 
30 months 3 (1.7) 1 (1.3) 
31 months 0 (0.0) 0 (0.0) 
32 months 1 (0.6) 0 (0.0) 


Note: Data include 673 participants in BCO programs and 566 participants in FOC programs for Job Status Improvement. Percentage of 


participants represents those who had a “job improvement” in 1-14 months after program enrollment. 


Data include 181 participants in BCO programs and 79 participants in FOC-only programs. Percentage of participants represents those whose 


maximum credit score after program enrollment was higher than their credit score at intake. 
Source: BCO and FOC administrative data in Salesforce: FFT™ Employment Record/Job Advancement Record. 
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Exhibit B-25. Financial Well-Being Outcome Measures Examined: Data Source and Measure Construction 


Measure Data Source | Definition/Coding 
Credit Score Increase at BCO and FOC | 1=Hasa credit score at baseline and follow-up, and the credit score at follow-up is greater than the credit score at 
Follow-Up baseline 
0 = Has a credit score at baseline and follow-up, and the credit score at follow-up is less than or equal to the credit score at 
baseline 
Credit Score Increase at Least | BCO and FOC | 1=Hasa credit score at baseline and follow-up, and the credit score level (A/B/C/D/E) at follow-up is greater than the 
One Level at Follow-Up credit score level at baseline 
0 = Has a credit score at baseline and follow-up, and the credit score level (A/B/C/D/E) at follow-up is less than or equal to 
the credit score level at baseline 
For all FICO scores — Transunion, Equifax, Experian: (A= 800-850, B= 740-799, C= 670-739, D= 580-669, E= 300-579) 
For all non-FICO scores — Transunion, Vantage: (A=781-850, B=720-780, C=658-719, D=601-657, E=300-600) 
Credit Score Reached a Score | BCO and FOC | 1=Hasa credit score at baseline and follow-up, and the credit score at follow-up is greater than or equal to 580 (580 is the 
of 580 at Follow-Up minimum credit score to receive a loan) 
0 Has a credit score at baseline and follow-up, and the credit score at follow-up is less than 580 
Credit Score Reached a Score | BCO and FOC | 1=Hasa credit score at baseline and follow-up, and the credit score at follow-up is greater than or equal to 620 (620 is the 
of 620 at Follow-Up minimum credit score for a conventional home loan approval) 
0 Has a credit score at baseline and follow-up, and the credit score at follow-up is less than 620 
Credit Score at Follow Up BCO and FOC | = Credit score at follow-up, for individuals with a credit score at baseline and a credit score at follow-up 
Debt Decrease at Follow-Up BCO and FOC | 1=Hasa measure of total debt at baseline and follow-up, and total debt at follow-up is less than total debt at baseline 
0 = Has a measure of total debt at baseline and follow-up, and total debt at follow-up is greater than or equal to total debt at 
baseline 
*Calculations for debt exclude student loans, mortgages, and transportation loans. 
Net Worth Increase at Follow- | BCO and FOC | 1 =Hasa measure of total net worth at baseline and follow-up, and total net worth at follow-up is greater than total net 
Up worth at baseline 
0 = Has a measure of total net worth at baseline and follow-up, and total net worth at follow-up is less than or equal to total 
net worth at baseline 
*Calculations for net worth exclude student loans, mortgages, and transportation loans. 
Change in Debt at Follow-Up BCO and FOC | = Difference between individual's total debt at follow-up and total debt at baseline, for individuals with measures of total 
debt at baseline and at follow up 
*Calculations for debt exclude student loans, mortgages, and transportation loans. 
Change in Net Worth at Follow- | BCO and FOC | = Difference between individual's total net worth at follow-up and total net worth at baseline, for individuals with measures 


Up 


of total net worth at baseline and at follow-up 
*Calculations for net worth exclude student loans, mortgages, and transportation loans. 


Source: BCO and FOC administrative data in Salesforce. 
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Exhibit B-26. Education Outcome Measure Examined: Data Source and Measure Construction 


Measure Data Source Definition/Coding 
Test Score Gain from Pre- to Post-test BCO = The difference between the final post-test TABE® Reading scale score and pre-test TABE® Reading scale score 


Exhibit B-27. Participant Characteristics at Baseline: Data Source and Measure Construction 


Measure Data Source Definition/Coding 
Gender BCO and FOC | 1=Female 
0=Male 
Race/Ethnicity BCO and FOC | 1=Black/African American non-Hispanic 
2=White non-Hispanic 
3=Hispanic 
4=Other (American Indian, Asian, Multiracial. Biracial, Other) 
Age at Intake BCO and FOC | 1=17-24 years old 


2=25-34 years old 

3=35-44 years old 

4=45-54 years old 

5=55+ years old 

Marital Status at Intake BCO and FOC | 1=Divorced 

2=Married 

3=Separated 

4=Single 

5=Widowed 

Primary Language Spoken BCO and FOC | 1=English 

0=Not English 

Level of Education at Intake BCO and FOC | 1=No High School Diploma/Equivalency 
2=High School Diploma or High School Equivalency 
3=Some College 

4=College Certificate or Degree 


Employed at Intake BCO and FOC | 1=Employed 
O=Not Employed 
FICO Credit Score Range at Intake BCO and FOC | 1=Exceptional (800+) 


2=Very Good (740-799) 

3=Good (670-739) 

4=Fair (580-669) 

5=Poor (<580) 

6=Insufficient Credit Score History 
7=Non-FICO Score Available 
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Estimating the Effects of BCO 
To account for the matching block design, Abt estimated the impact of BCO using regression models that 


include terms for the matching blocks. The inclusion of terms for the matching blocks ensures that the 
overall estimate of the treatment-comparison group outcome difference is calculated by comparing the 
BCO and FOC participant outcomes within the matched blocks and calculating a precision-weighted 
average of the within-block differences to produce the overall estimate. To control for other baseline 
differences between the groups that might be related to outcomes, the model also includes covariates for 
participants’ baseline level of education, employment, credit score range, gender, age, and race/ethnicity. 


For each outcome, we estimated a model that reflects this matched design and has the structure of 
Equation 2. 


Eq(2)) Y= Bo + BiTi + Diet An Xi + Lihat YomDm + & 
where: 


Y; is the outcome of interest (e.g., receipt of educational credential) for the i individual in the m™ 
matching block. 


T;is an indicator variable equal to 1 if individual i was an BCO participant and 0 if a FOC participant. 


Xx; is the k baseline covariate; these include baseline level of education, employment, credit score 
range, gender, age, and race/ethnicity. 


Dyn is the indicator variable representing the m matching block. 
€; is the usual random error term. 


In this model, 61 represents the impact of BCO on the study outcomes, where the impact estimate is 
averaged over the matching blocks. The p-values reported for impact estimates are two-tailed to account 
for the possibility that the intervention might adversely affect one or more of the outcomes. The 
coefficients on the covariates (A;,) reflect the relationship between the outcome measure and each of the 
covariates while controlling for others. 


Pre-Post Reading Gain Analysis 
Analysis of Changes in BCO Participants’ Reading Skills 
For each reading test, Abt converted BCO participants’ raw scores into two measures: (1) grade 


equivalent scores that are useful for descriptive purposes; and (2) scale scores that are appropriate for 
more complex statistical analyses. Change scores that represent the amount of a BCO participant’s skill 
change from pre- to post-tests were constructed by subtracting each participant’s pre-test scale score from 
their post-test scale scores. The statistical significance of the mean scale score gain was assessed using the 
paired t-test. 


Hypothesis Testing of Change in Scores 
Abt tested the null hypothesis that participants’ change scores are zero by conducting a paired t-test. The 


null hypothesis for the paired t-test is that the average population gain is zero scale score points. The two- 
tail alternative hypothesis is that the average gain is not equal to zero. The use of the paired t-test is 
predicated on the assumption that, absent BCO, the reading ability of the learners would not be expected 
to measurably improve over the span of time from pre-test to post-test. If the average change is positive 
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and the null hypothesis is rejected, we interpret this as an indication that program participation is 
associated with the shift in the distribution of reading skills among the population of participants. 


Effect Sizes 
Abt examined the effect size of the pre-post changes in BCO participants’ reading test scores. The effect 


size is a measure of the mean gain expressed in standard deviation units. An advantage of expressing 
mean gain as an effect size is that the mean gains for each of the various standardized reading tests can be 
directly compared to one another. This is possible because, when expressed as an effect size, they are 
each expressed in a common metric. 


Additionally, there is a commonly used rule-of-thumb for evaluating the magnitude of effect sizes 
(Cohen, 1988). For effect sizes calculated as differences in means, effect sizes of .20 or lower are 
generally considered to be "small"; a "medium" effect size is approximately .50; and a “large” effect size 
is .80 or greater. /n interpreting skill change data, we used as conservative standard by focusing on skill 
gains that were statistically significant (by paired t-tests at p<.05 level) and that produced effect sizes of 
.20 or higher. 


Effect sizes were calculated using the formula shown below: 


Effect Size pox:-pre = 


post—pre 


Where: 


X __ is the standard score mean at post-test, 


post 


X is the standard score mean at pre-test, 
he 


ar is the pooled standard deviation of the pre- and post-test means. 


Because the sample sizes are equal for the pre- and post-tests, the pooled standard deviation was 
calculated as 


pooled 


where § and § vost HEE the standard deviations for the pre- and post-test data. 
pre OS) 


Additionally, effect sizes were calculated for pre-follow-up gains and post-follow-up gains. To ensure 
that these effect sizes were directly comparable to the pre-post effect sizes, Abt used the same value for 
the pooled standard deviation in the denominator for the calculations of the pre-follow-up and post- 
follow-up effect sizes, as was used for the pre-post gains. 
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The formulas are shown below. 


X —X X -X 
. = followup pre _ followup post 
Effect SIZE pstowup-pre s 9 Effect SIZE rotowup-post Ss : 


pooled pooled 


where Spooiea is the same as shown above. 


Treatment of Missing Data 
Abt did not impute any baseline or outcome data. By design, only participants with non-missing baseline 
data were included in the matched sample. All analyses included participants with non-missing outcomes 
(complete case analysis). 


Multiple Outcome Measures 
To address multiple comparisons, Abt defined confirmatory and exploratory analysis components prior to 


data analysis (Schochet, 2008). Because there is a single confirmatory impact estimate for the 
evaluation’s outcome domains of employment and financial well-being, no multiple comparison 
corrections are required to reduce the chance of a Type-I error. Because the purpose of exploratory 
analyses is to examine relationships within the data—for example, to identify hypotheses for more 
rigorous future evaluation, Abt did not conduct multiplicity adjustments. Instead, we included explicit 
statements in Appendix C that these exploratory analyses do not provide rigorous evidence on the 
intervention’s overall effectiveness. 
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Appendix C: Supplemental Tables and Information on Study 
Findings 


This appendix provides the results of the exploratory analyses concerning employment and financial well- 
being that Abt conducted in the BCO evaluation’s impact study. Note that these exploratory analyses do 
not provide rigorous evidence on the intervention’s overall effectiveness. 


The appendix also includes BCO site-level data for the TABE® pre-post test results for each BCO site, 
which was one of the treatment group’s descriptive education outcomes measured used in the study. 


Employment 


The impact study’s exploratory research questions related to employment and results from the analyses to 
address those questions are described below. 


Exploratory Research Questions for Employment 
1. Were BCO participants more likely to increase their hourly wages after program entry relative to a 
comparison group who received only FOC services? 


2. Were BCO participants more likely to be placed in jobs with higher hourly wages relative to a 
comparison group who received only FOC services? 


3. Were BCO participants more likely to work more hours per week after program entry relative to a 
comparison group who received only FOC services? 


Impacts on Employment 

Presented in Exhibit C-1 are the findings from the employment impact analyses. The far-left column in 
the table lists the employment outcomes that are reported. The second column contains the outcomes for 
the treatment group and the third column contains the outcomes for the comparison group. The fourth 
column lists the impact results. The fifth column lists the standard error and the sixth column provides the 
p-Value for each impact. The first row in the exhibit has data for the confirmatory research question, 
which is discussed in the main body of the report. The data in the other rows in the table pertain to the 
exploratory research questions. Because the confirmatory employment measure was observed over a 
period of months, the study also conducted an exploratory analysis that included the number of months 
that elapsed between enrollment and measurement of the confirmatory outcome as a covariate in the 
impact model to investigate the sensitivity of the results to the timing of the measure. Exhibit C-2 shows 
that the finding remains the same. 


Wage Increase 

Abt analyzed two types of wage increases to address exploratory Research Questions 1 and 2 concerning 
hourly wages: (1) the maximum hourly wage increase for participants in months 1-14 after enrollment, 
and (2) the latest hourly wage increase for participants in months 1-14 after enrollment. The analyses of 
participants’ maximum hourly wage data indicated that a slightly higher percentage of BCO treatment 
group participants (87.5%) increased their hourly wages compared to FOC comparison group participants 
(85.2%), but this difference is not statistically significant. A similar pattern was found in the analyses of 
participants’ latest hourly wage after enrollment. 


Abt analyzed participants’ mean hourly wage increase in months 1-14 after enrollment. Exhibit C-1 
shows that FOC comparison group participants’ maximum mean hourly wage increase ($11.80) was 
significantly higher than the maximum mean hourly wage increase for BCO treatment group participants 
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($10.38), with an impact of -$1.42 for the treatment group. Analyses of the latest mean hourly wage 
increase indicated the same pattern of results. 


We also analyzed participants’ mean hourly wage in months 1-14 after enrollment. Exhibit C-1 shows 
that FOC comparison group participants’ maximum mean hourly wage ($15.00) was higher than the 
maximum mean hourly wage for BCO treatment group participants ($13.23), with an impact of -$1.77 for 
the treatment group. Analyses of the latest mean hourly wage showed the same pattern of results. 


Hours Worked 

Research Question 3 explored whether participants increased the number of hours they worked in a week 
in months 1-14 after enrollment. Exhibit C-1 shows that BCO treatment group participants (92.4%) were 
slightly more likely than comparison group participants (91.3%) to increase the number of hours that they 
worked in a week, but this increase is not statistically significant. We also examined the number of hours 
that participants worked during months 1-14 after enrollment and found that comparison group 
participants increased their number of hours worked in a week by 29.92 hours compared with treatment 
group participants, who increased the number of hours worked in a week by 28.72. However, this increase 
is not statistically significant. 


The final analysis concerning hours worked examined the number of hours per week that participants 
worked in months 1-14 after enrollment. Exhibit C-1 shows that FOC comparison group participants 
worked a mean of 34.29 hours per work compared to BCO treatment group participants who worked a 
mean of 33.20 hours per week. This difference is not statistically significant. 


Exhibit C-1. Impacts on Confirmatory and Exploratory Employment Outcomes 


|={010) FOC SIETOET«C| 


Participants Participants Impact Error p-Value 4 


Job Status (Confirmatory) 


Job Status Improvement 


Wage (Maximum Wage at Follow-up) (Exploratory) 


55.6% 


36.8% 


18.7% 


3.0 


.000*** 


Wage Increase 87.5% 85.2% 2.3% 2.0 251 
Mean Hourly Wage Increase 10.38 11.80 -1.42 0.41 .001** 
Mean Hourly Wage at Follow-up 13.23 15.00 -1.77 0.39 .000*** 
Wage (Latest Wage at Follow-up) (Exploratory) 

Wage Increase 86.5% 85.3% 1.2% 2.0 O37 
Mean Hourly Wage Increase 10.02 11.85 -1.83 0.39 .000*** 
Mean Hourly Wage at Follow-up 12.89 14.83 -1.93 0.36 .000*** 


Hours Worked (Latest Hours Worked at Follow-up) (Exploratory) 


Hours Worked Increase 92.4% 91.3% 1.1% 1.7 523 
Mean Hours Worked Increase 28.72 29.92 -1.19 0.99 .230 
Hours Worked per Week at Follow-up 33.20 34.29 -1.09 0.83 187 


2 p-Values shown in this column are for tests of whether there was a statistically significant impact for the outcome in the row. 


Notes: 


* Difference is statistically significant at the .05 level. 


™ Difference is statistically significant at the .01 le 
** Difference is statistically significant at the .001 
Sample Sizes: 


vel. 
level. 


Job Status Improvement: BCO Participants=673 clients, FOC Participants=566 clients. 
Wage Increase: BCO Participants=407 clients, FOC Participants=288 clients. 


Mean Hourly Wage Increase: BCO Participants=407 clients, FOC Participants=288 clients. 
Mean Hourly Wage at Follow-up: BCO Participants=523 clients, FOC Participants=310 clients. 
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Hours Worked Increase: BCO Participants=367 clients, FOC Participants=269 clients. 

Mean Hours Worked Increase: BCO Participants=367 clients, FOC Participants=269 clients. 

Hours Worked per Week at Follow-up: BCO Participants=484 clients, FOC Participants=291 clients. 

Source: BCO and FOC administrative records in Salesforce FFT™ Employment /Employment Advancement Record, FFT™ Employment 
Counseling Record. 


Exhibit C-2. Sensitivity Analysis: Impact on Confirmatory Employment Outcome Controlling for 
Number of Months Between Enrollment and Measure of Outcome 


BCO FOC Impact Standard p-Value @ 
Participants Participants Error 


Job Status (Confirmatory) 
Job Status Improvement 55.6% 40.7% 14.8% 3.0 .000*** 


Note: ***Difference is statistically significant at the .001 level. Data include 673 participants in BCO programs and 566 participants in FOC 
programs. Percentage of participants represents those who had a “job improvement’ in 1-14 months after program enrollment. 
Source: BCO and FOC administrative data in Salesforce: FFT™ Employment Record/Job Advancement Record. 


Interpretation of Results 

The overall results concerning BCO treatment and FOC comparison groups’ wage increases show that 
most participants in both groups increased their wages from the time of their enrollment in the BCO or 
FOC program to follow-up, and both groups increased the amount of their mean hourly wages. Both 
groups also increased their mean hourly wage at follow-up. The fact that both BCO treatment and FOC 
comparison group participants received employment services may account for the large percentage of 
both groups who increased their overall wages and mean hourly wages. The value of participating in 
occupational training and earning occupational credentials did not have an effect during this short-term 
follow-up period. 


One factor that might explain why the treatment group’s mean hourly wage at follow-up is lower than the 
FOC comparison group’s is that over half (55%) of the BCO treatment group obtained jobs in the 
supportive healthcare sector, where the hourly rate of pay is often at minimum wage. In contrast, over half 
(59%) of FOC comparison group participants obtained jobs in building and grounds, food preparation, 
and office and administrative support sectors where the hourly rate of pay might have been higher than 
jobs in healthcare. 


Financial Well-Being 


The impact study’s exploratory research questions related to financial well-being and results from the 
analyses to address those questions are described below. 


Exploratory Research Questions for Financial Well-Being 
1. Are BCO participants more likely to have lower overall debt after program entry relative to a 
comparison group who received only FOC services? 


2. Are BCO participants more likely to have increased their net worth after program entry relative to a 
comparison group who received only FOC services? 


Impacts on Financial Well-Being 

Presented in Exhibit C-3 are the findings from the financial well-being impact analyses. The far-left 
column in the table lists the financial well-being outcomes that are reported. The second column contains 
the outcomes for the treatment group and the third column contains the outcomes for the comparison 
group. The fourth column lists the impact results. The fifth column lists the standard error, and the six 
column provides the p-Value for each impact. Because the confirmatory financial well-being measure was 
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observed over a period of months, the study also conducted an exploratory analysis that include the 
number of months that elapsed between enrollment and measurement of the confirmatory outcome as a 
covariate in the impact model to investigate the sensitivity of the results to the timing of the measure. 
Exhibit C-4 shows that the finding remains the same. 


Total Debt 
Abt examined participants who had a measure of total debt at baseline and follow-up and analyzed 


whether their total debt was lower at follow up than at baseline. For the debt analysis, the calculations 
excluded student loans, mortgages, and transportation loans, which has been LISC’s practice in previous 
studies. 


Exhibit C-3 shows that comparison group participants (57%) were more likely to decrease their debt than 
treatment group participants (29%), with an impact of -28% for the treatment group. Comparison group 
participants also decreased their debt by $3,057.70 while treatment group participants increased their debt 
by $1,785.17. 


Net Worth 

Abt also examined participants who had a measure of net worth at baseline and follow-up and analyzed 
whether their total net worth was higher at follow up than at baseline. Exhibit C- 3 shows while both 
treatment and comparison group participants had increased their net worth, a larger percentage of 
comparison group participants (65%) increased their net worth compared with treatment group 
participants (41%). Comparison group participants’ mean change in net worth was $9,954.86 compared 
with treatment group participants’ mean change in net worth, which was $2,636.26. 


Exhibit C-3. Impacts on Confirmatory and Exploratory Financial Well-Being Outcomes 


|={010) FOC Standard 


Participants Participants Impact Error p-Value @ 


Credit Score (Maximum Credit Score at Follow-up) (Confirmatory) 


Credit Score Increase 68.0% 79.8% -11.8% 7.1 100 
Credit Score Level Increase 28.7% 36.3% -7.5% 74 310 
Credit Score Reached 580 59.7% 67.7% -8.0% 7.3 212 
Credit Score Reached 620 35.4% 42.7% -74% 5.6 192 
Credit Score Value at Follow-Up 604.1 607.9 -3.8 8.1 637 


Credit Score (Latest Credit Score at Follow-Up) (Confirmatory) 


Credit Score Increase 54.8% 16.7% -21.9% 76 .005** 
Credit Score Level Increase 23.7% 32.8% -9.1% 7.1 .202 
Credit Score Reached 580 49.7% 63.1% -13.4% 75 .076 
Credit Score Reached 620 31.1% 36.3% -5.3% 5.4 333 
Credit Score Value at Follow-Up 591.5 605.7 -14.2 9.0 17 


Financial Worth » (Exploratory) 


Debt Decrease 29.0% 57.0% -28.0% 7.5 .000*** 
Net Worth Increase 41.0% 65.0% -24.0% 7.9 .002** 
Change in Debt 1,785.17 -3,057.70 4,842.86 1,603.53 .003** 
Change in Net Worth 2,636.26 9,954.86 -7,318.60 4,640.87 A17 


a p-Values shown in this column are for tests of whether there was a statistically significant impact for the outcome in the row. 
> Calculations for debt and net worth exclude student loans, mortgages, and transportation loans. 

Notes: 

* Difference is statistically significant at the .05 level. 

*™* Difference is statistically significant at the .01 level. 
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** Difference is statistically significant at the .001 level. 

Sample Sizes: 

Credit Score (Maximum Credit Score at Follow-Up): BCO Participants=181 clients, FOC Participants=79 clients. 
Credit Score (Latest Credit Score at Follow-Up): BCO Participants=177 clients, FOC Participants=79 clients. 
Financial Worth: BCO Participants=194 individuals, FOC Participants=81clientss. 

Source: BCO and FOC administrative data in Salesforce: FFT™ Credit Report Record, FFT™ Balance Sheet Record. 


Exhibit C-4. Sensitivity Analysis: Impact on Confirmatory Financial Well-Being Outcome 
Controlling for Number of Months Between Enrollment and Measurement of Outcome 


BCO FOC Standard 


Participants Participants Impact Error p-Value @ 
Credit Score (Confirmatory) 


Credit Score (Maximum Credit Score 68.0% 84.5% 16.5% 018" 
at Follow-Up) Increase 


Note: Data include 181 participants in BCO programs and 79 participants in FOC-only programs. Percentage of participants represents those 
whose maximum credit score after program enrollment was higher than their credit score at intake. 
Source: BCO and FOC administrative data in Salesforce: FFT™ Credit Score Record. 


Interpretation of Results 

The FOC comparison group participants’ increases in net worth and decreases in debt might be explained 
by the length of time since they have been in the workforce since enrolling in the FOC compared with 
BCO treatment group participants. The comparison group participants’ length of time in the workforce 
along with their possibly higher hourly wage rates might have enabled them to reduce their debt in the 
near term compared to treatment group participants. 


Education 


Abt’s evaluation plan included an exploratory research question for the treatment group concerning 
whether BCO participants were likely to increase their reading skills (as measured by a standardized test) 
after participating in the BCO program. To address this question, Abt conducted a pre-post analysis of all 
treatment group participants’ performance on the Test for Adult Basic Education (TABE®) Reading Test, 
not just individuals included in the matched samples. Exhibit 12 in the main body of the report shows the 
results from the pre-post analysis of the TABE® Reading scores for the whole treatment group. Exhibit C- 
5 shows the results of the pre-post analysis of TABE® Reading test scores by BCO treatment site. The 
sites varied in the amount of complete pre-post data that were entered into the Salesforce database, with 
two sites submitting minimal data. Four of the sites had statistically significant gains in reading test scores 
with small effects. 
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Number of Learners 


All BCO 
Sites 


(n=282) 


Site A 


Site B 


Site 
Cc 


(n=4) (n=36) _(n=6) 


APPENDIX C: Supplemental Tables and Information 


on Study Findings 


STi) 
) 
(n=1) 


Site E 
Me) 


SiGag 
(n=12) 


Site G 
(n=68) 


Average Test Score Gain from Pre- 9.05 19.00 | -12.81 | -4.67 | 13.00 | 8.98 8.75 21.40 
to Final Post Test 

Standard Deviation of Pre-Test 65.99 19.50 | 54.79 | 50.42 48.75 | 31.33 60.03 
Standardized Test Score Gain 0.14 0.97 -0.23 | -0.09 0.18 0.28 0.36 
p-Value of Gain from Paired T-Test 0.0002*** | 0.004** | 0.078 | 0.460 0.001** | 0.589 | 0.0004*** 


Notes: 


* Difference is statistically significant at the .05 level. 

*™* Difference is statistically significant at the .01 level. 
** Difference is statistically significant at the .001 level. 
Source: BCO administrative data in Salesforce: FFT™ TABE® Records. 


Exhibit C-6 shows the distribution of BCO sites’ comparison group participants’ reading skills by grade 
level-equivalence at baseline. A slightly higher percentage of participants scored below the 6" grade 
equivalent at baseline. This finding is likely due to the large number of participants at Site D and Site E, 
whose native language is not English. The treatment group participants are distributed across the four 
grade-equivalent levels, which reflects the BCO treatment sites’ skill criteria for participating in the 
occupational training that they offered. Sites generally enrolled participants who tested at the 6" grade 
equivalent or higher at program intake. 


Exhibit C-6. TABE® Reading Pre-Test Results for BCO Participants at Baseline by Site 


All BCO 
Sites 


(n=1,133) 


Site A 
(n=217) 


Site B 
(n=117) 


Site C 
(ew!) 


Site D 
(n=146) 


Site E 
(n=354) 


Nice 
(i 24) 


Site G 
(n=145) 


Reading (n=594) (n=83) (n=110) (n=73) (ie) (iw A) ‘(ie :)) (n=92) 
Grade Equivalent 29.6 6.0 15.5 19.2 8.5 724 0.0 14.1 
<5.9 (%) 

Grade Equivalent 24.9 20.5 31.8 32.9 19.2 18.2 26.3 29.4 
6.0-8.9 (%) 

Grade Equivalent 19.9 20.1 25.5 26.0 19.2 7.1 26.3 22.8 
9.0-10.9 (%) 

Grade Equivalent 25.6 45.8 27.3 21.9 53.2 1.8 47.4 33.7 
11-12.9 (%) 


Source: BCO administrative data in Salesforce: FFT™ TABE® Record. 
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